From a63fc037806d51960edbf4d76f3cb521df4f715e Mon Sep 17 00:00:00 2001 From: Adam Fletcher Date: Thu, 5 Oct 2023 09:15:11 -0700 Subject: [PATCH 1/7] add RU calcuation to explain docs --- .../sql-statement-explain-analyze.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index 3954c9ed4dbc0..3bc38b52fc6bd 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -294,6 +294,33 @@ RU:273.842670 > > This value shows the actual RUs consumed by this execution. The same SQL statement might consume different amounts of RUs each time it is executed due to the effects of caching (for example, [coprocessor cache](/coprocessor-cache.md)). +The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; specifically, the `executeInfo` block. For example, given: + +``` + 'executeInfo': 'time:2.55ms, loops:2, RU:0.329460, Get:{num_rpc:1, ' + 'total_time:2.13ms}, total_process_time: 231.5µs, ' + 'total_wait_time: 732.9µs, tikv_wall_time: 995.8µs, ' + 'scan_detail: {total_process_keys: 1, total_process_keys_size: ' + '150, total_keys: 1, get_snapshot_time: 691.7µs, rocksdb: ' + '{block: {cache_hit_count: 2, read_count: 1, read_byte: 8.19 ' + 'KB, read_time: 10.3µs}}}', +``` + +The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the +calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) + +If you are using 7.1.x, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is : + +``` +before : + consumption.RRU += float64(kc.ReadBaseCost) -> kv.ReadBaseCost * rpc_nums +after: + consumption.RRU += float64(kc.ReadBytesCost) * readBytes -> kc.ReadBytesCost * total_process_keys_size + consumption.RRU += float64(kc.CPUMsCost) * kvCPUMs -> kc.CPUMsCost * total_process_time +``` + +For writes & batch gets the calculation is similar with different base values. + ### Other common execution information The Coprocessor operators usually contain two parts of execution time information: `cop_task` and `tikv_task`. `cop_task` is the time recorded by TiDB, and it is from the moment that the request is sent to the server to the moment that the response is received. `tikv_task` is the time recorded by TiKV Coprocessor itself. If there is much difference between the two, it might indicate that the time spent waiting for the response is too long, or the time spent on gRPC or network is too long. From 1c2b3767ce349c69e966f2ab7d0db7a8fed47983 Mon Sep 17 00:00:00 2001 From: Adam Fletcher Date: Thu, 5 Oct 2023 15:21:30 -0700 Subject: [PATCH 2/7] fix line break --- sql-statements/sql-statement-explain-analyze.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index 3bc38b52fc6bd..85c7810af803f 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -306,10 +306,9 @@ The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; speci 'KB, read_time: 10.3µs}}}', ``` -The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the -calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) +The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) -If you are using 7.1.x, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is : +If you are using 7.1.x, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: ``` before : From 8dd01a6399dfe8fe7a3777736def102d176836a2 Mon Sep 17 00:00:00 2001 From: Adam Fletcher <135756383+adamf-db@users.noreply.github.com> Date: Fri, 6 Oct 2023 08:21:11 -0700 Subject: [PATCH 3/7] Update sql-statements/sql-statement-explain-analyze.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Daniël van Eeden --- .../sql-statement-explain-analyze.md | 32 +++++++++++++++---- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index 85c7810af803f..cf2916ed01828 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -297,13 +297,31 @@ RU:273.842670 The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; specifically, the `executeInfo` block. For example, given: ``` - 'executeInfo': 'time:2.55ms, loops:2, RU:0.329460, Get:{num_rpc:1, ' - 'total_time:2.13ms}, total_process_time: 231.5µs, ' - 'total_wait_time: 732.9µs, tikv_wall_time: 995.8µs, ' - 'scan_detail: {total_process_keys: 1, total_process_keys_size: ' - '150, total_keys: 1, get_snapshot_time: 691.7µs, rocksdb: ' - '{block: {cache_hit_count: 2, read_count: 1, read_byte: 8.19 ' - 'KB, read_time: 10.3µs}}}', + 'executeInfo': + time:2.55ms, + loops:2, + RU:0.329460, + Get:{ + num_rpc:1, + total_time:2.13ms + }, + total_process_time: 231.5µs, + total_wait_time: 732.9µs, + tikv_wall_time: 995.8µs, + scan_detail: { + total_process_keys: 1, + total_process_keys_size: 150, + total_keys: 1, + get_snapshot_time: 691.7µs, + rocksdb: { + block: { + cache_hit_count: 2, + read_count: 1, + read_byte: 8.19 KB, + read_time: 10.3µs + } + } + }, ``` The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) From 662f55ae3ee53120b87f22662ff1661be0534a7c Mon Sep 17 00:00:00 2001 From: Adam Fletcher Date: Fri, 6 Oct 2023 08:22:35 -0700 Subject: [PATCH 4/7] PR feedback --- sql-statements/sql-statement-explain-analyze.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index cf2916ed01828..b29602ec38f8d 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -326,12 +326,13 @@ The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; speci The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) -If you are using 7.1.x, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: +If you are using 7.1+, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: ``` -before : +before key/value request is processed: consumption.RRU += float64(kc.ReadBaseCost) -> kv.ReadBaseCost * rpc_nums -after: + +after key/value request is processed: consumption.RRU += float64(kc.ReadBytesCost) * readBytes -> kc.ReadBytesCost * total_process_keys_size consumption.RRU += float64(kc.CPUMsCost) * kvCPUMs -> kc.CPUMsCost * total_process_time ``` From 562c11550247d1f28dfb2f6d38b011ed19c2a6d8 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 17 Oct 2023 15:35:22 +0800 Subject: [PATCH 5/7] Update sql-statements/sql-statement-explain-analyze.md --- sql-statements/sql-statement-explain-analyze.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index b29602ec38f8d..92bf536146128 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -337,7 +337,7 @@ after key/value request is processed: consumption.RRU += float64(kc.CPUMsCost) * kvCPUMs -> kc.CPUMsCost * total_process_time ``` -For writes & batch gets the calculation is similar with different base values. +For writes and batch gets, the calculation is similar with different base values. ### Other common execution information From 3dd232d42e92f58218b52629805662ae5732e42b Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 17 Oct 2023 16:09:55 +0800 Subject: [PATCH 6/7] Update sql-statements/sql-statement-explain-analyze.md --- sql-statements/sql-statement-explain-analyze.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index 92bf536146128..731f91e410905 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -326,7 +326,7 @@ The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; speci The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) -If you are using 7.1+, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: +If you are using TiDB v7.1, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: ``` before key/value request is processed: From aab8d28fa68c8f7558735ff722214ebd463f435f Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 17 Oct 2023 20:58:33 +0800 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: Aolin --- .../sql-statement-explain-analyze.md | 62 +++++++++---------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/sql-statements/sql-statement-explain-analyze.md b/sql-statements/sql-statement-explain-analyze.md index 731f91e410905..462b0c8d84ec9 100644 --- a/sql-statements/sql-statement-explain-analyze.md +++ b/sql-statements/sql-statement-explain-analyze.md @@ -294,39 +294,39 @@ RU:273.842670 > > This value shows the actual RUs consumed by this execution. The same SQL statement might consume different amounts of RUs each time it is executed due to the effects of caching (for example, [coprocessor cache](/coprocessor-cache.md)). -The RU count can be calculated from the other values in `EXPLAIN ANALYZE`; specifically, the `executeInfo` block. For example, given: - -``` - 'executeInfo': - time:2.55ms, - loops:2, - RU:0.329460, - Get:{ - num_rpc:1, - total_time:2.13ms - }, - total_process_time: 231.5µs, - total_wait_time: 732.9µs, - tikv_wall_time: 995.8µs, - scan_detail: { - total_process_keys: 1, - total_process_keys_size: 150, - total_keys: 1, - get_snapshot_time: 691.7µs, - rocksdb: { - block: { - cache_hit_count: 2, - read_count: 1, - read_byte: 8.19 KB, - read_time: 10.3µs - } - } - }, +You can calculate the RU from the other values in `EXPLAIN ANALYZE`, specifically the `execution info` column. For example: + +```json +'executeInfo': + time:2.55ms, + loops:2, + RU:0.329460, + Get:{ + num_rpc:1, + total_time:2.13ms + }, + total_process_time: 231.5µs, + total_wait_time: 732.9µs, + tikv_wall_time: 995.8µs, + scan_detail: { + total_process_keys: 1, + total_process_keys_size: 150, + total_keys: 1, + get_snapshot_time: 691.7µs, + rocksdb: { + block: { + cache_hit_count: 2, + read_count: 1, + read_byte: 8.19 KB, + read_time: 10.3µs + } + } + }, ``` -The base costs are in [the pd source](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are done in [model.go](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) +The base costs are defined in the [`tikv/pd` source code](https://github.com/tikv/pd/blob/aeb259335644d65a97285d7e62b38e7e43c6ddca/client/resource_group/controller/config.go#L58C19-L67) and the calculations are performed in the [`model.go`](https://github.com/tikv/pd/blob/54219d649fb4c8834cd94362a63988f3c074d33e/client/resource_group/controller/model.go#L107) file. -If you are using TiDB v7.1, the calculation is sum `BeforeKVRequest() + AfterKVRequest()` in `pd/pd-client/model.go`, that is: +If you are using TiDB v7.1, the calculation is the sum of `BeforeKVRequest()` and `AfterKVRequest()` in `pd/pd-client/model.go`, that is: ``` before key/value request is processed: @@ -337,7 +337,7 @@ after key/value request is processed: consumption.RRU += float64(kc.CPUMsCost) * kvCPUMs -> kc.CPUMsCost * total_process_time ``` -For writes and batch gets, the calculation is similar with different base values. +For writes and batch gets, the calculation is similar with different base costs. ### Other common execution information