infoschema/metric_schema: add node cpu/mem/net load metric tables #14670

crazycs520 · 2020-02-07T07:13:09Z

What problem does this PR solve?

add node cpu/mem/net load metric tables.

eg:

> select * from `METRIC_SUMMARY` where metric_name like "node%";
+-------------------------------------+----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
| METRIC_NAME                         | TIME                       | SUM_VALUE                   | AVG_VALUE                   | MIN_VALUE                   | MAX_VALUE                   |
+-------------------------------------+----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
| node_kernel_context_switches        | 2020-02-07 14:57:05.871000 |      655519.545634          |       65551.9545634         |       64829.2               |       66990.0666667         |
| node_kernel_interrupts              | 2020-02-07 14:57:05.871000 |      285098.561633          |       28509.8561633         |       24890.6               |       36146.0921354         |
| node_disk_io_util                   | 2020-02-07 14:57:05.871000 |           0.0061111293834   |           0.000152778234585 |           0.0               |           0.000822222222471 |
| node_cpu_usage                      | 2020-02-07 14:57:05.871000 |         983.351640034       |          12.2918955004      |           0.0               |          95.0610013553      |
| node_virtual_cpus                   | 2020-02-07 14:57:05.871000 |         400.0               |          40.0               |          40.0               |          40.0               |
| node_disk_throughput                | 2020-02-07 14:57:05.871000 |     4568739.61143           |      114218.490286          |           0.0               |      451955.463698          |
| node_total_memory_swap              | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_memory_buffers                 | 2020-02-07 14:57:05.871000 |  8409874432.0               |   840987443.2               |   840986624.0               |   840990720.0               |
| node_memory_dirty                   | 2020-02-07 14:57:05.871000 |     1277952.0               |      127795.2               |       65536.0               |      217088.0               |
| node_network_out_packets            | 2020-02-07 14:57:05.871000 |        5068.3477909         |          20.2733911636      |           0.0               |         309.748672252       |
| node_file_descriptor_allocated      | 2020-02-07 14:57:05.871000 |       85440.0               |        8544.0               |        8400.0               |        8800.0               |
| node_load15                         | 2020-02-07 14:57:05.871000 |          16.02              |           1.602             |           1.31              |           2.02              |
| node_disk_available_size            | 2020-02-07 14:57:05.871000 |           1.02136396513e+13 |           1.45909137876e+11 |      305152.0               |           4.37057007616e+11 |
| node_memory_active                  | 2020-02-07 14:57:05.871000 |           5.00219334656e+11 | 50021933465.6               | 49960075264.0               | 50104033280.0               |
| node_segments_retransmitted         | 2020-02-07 14:57:05.871000 |           0.244446419863    |           0.0244446419863   |           0.0               |           0.155559012422    |
| node_uptime                         | 2020-02-07 14:57:05.871000 |   192730650.591             |    19273065.0591            |    19272795.0582            |    19273335.0592            |
| node_disk_write_latency             | 2020-02-07 14:57:05.871000 |           0.00145512490499  |           7.27562452495e-05 |           0.0               |           0.000188208616744 |
| node_load1                          | 2020-02-07 14:57:05.871000 |          24.92              |           2.492             |           1.18              |           5.19              |
| node_memory_writeback_tmp           | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_filesystem_space_used          | 2020-02-07 14:57:05.871000 |        3347.76157822        |          47.8251654031      |           0.0               |          99.8465775993      |
| node_memory_available               | 2020-02-07 14:57:05.871000 |           1.09862304563e+12 |           1.09862304563e+11 |           1.097805824e+11   |           1.09930532864e+11 |
| node_kernel_forks                   | 2020-02-07 14:57:05.871000 |          27.377811365       |           2.7377811365      |           2.39994666785     |           3.93324592787     |
| node_network_utilization_out_hourly | 2020-02-07 14:57:05.871000 | 17404865994.3               |  1740486599.43              |  1740248889.04              |  1741354967.99              |
| node_tcp_in_use                     | 2020-02-07 14:57:05.871000 |         586.0               |          58.6               |          58.0               |          60.0               |
| node_network_out_errors             | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_process_open_fd_count          | 2020-02-07 14:57:05.871000 |        4451.0               |         148.366666667       |           7.0               |         428.0               |
| node_disk_size                      | 2020-02-07 14:57:05.871000 |           3.6101607168e+13  |           5.15737245257e+11 |   198896640.0               |           1.57323309875e+12 |
| node_network_utilization_in_hourly  | 2020-02-07 14:57:05.871000 | 13309315966.2               |  1330931596.62              |  1330581288.7               |  1332042573.38              |
| node_network_in_errors              | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_disk_state                     | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_processes_blocked              | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_memory_writeback               | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_network_in_packets             | 2020-02-07 14:57:05.871000 |        4958.83630643        |          19.8353452257      |           0.0               |         305.526543855       |
| node_disk_read_latency              | 2020-02-07 14:57:05.871000 | <null>                      | <null>                      | <null>                      | <null>                      |
| node_tcp_connections                | 2020-02-07 14:57:05.871000 |        1121.0               |         112.1               |         107.0               |         114.0               |
| node_network_out_drops              | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_network_interface_speed        | 2020-02-07 14:57:05.871000 |       50000.0               |         200.0               |           0.0               |        1000.0               |
| node_memory_cached                  | 2020-02-07 14:57:05.871000 |           8.79347232768e+11 | 87934723276.8               | 86624690176.0               | 89831473152.0               |
| node_load5                          | 2020-02-07 14:57:05.871000 |          20.23              |           2.023             |           1.42              |           2.92              |
| node_network_in_drops               | 2020-02-07 14:57:05.871000 |           0.0               |           0.0               |           0.0               |           0.0               |
| node_memory_mapped                  | 2020-02-07 14:57:05.871000 |  4651339776.0               |   465133977.6               |   463855616.0               |   468094976.0               |
| node_total_memory                   | 2020-02-07 14:57:05.871000 |           1.34905430016e+12 |           1.34905430016e+11 |           1.34905430016e+11 |           1.34905430016e+11 |
| node_memory_shared                  | 2020-02-07 14:57:05.871000 | 44279304192.0               |  4427930419.2               |  4426252288.0               |  4434640896.0               |
| node_memory_free                    | 2020-02-07 14:57:05.871000 |           2.36710756352e+11 | 23671075635.2               | 21735051264.0               | 24982966272.0               |
| node_memory_inactive                | 2020-02-07 14:57:05.871000 |           5.6814526464e+11  | 56814526464.0               | 55505920000.0               | 58710208512.0               |
| node_disk_iops                      | 2020-02-07 14:57:05.871000 |         223.77829734        |           5.5944574335      |           0.0               |          21.2449165537      |
| node_network_out_traffic            | 2020-02-07 14:57:05.871000 |     4852941.17995           |       19411.7647198         |           0.0               |      204910.022222          |
| node_processes_running              | 2020-02-07 14:57:05.871000 |         234.0               |          23.4               |           9.0               |          61.0               |
| node_network_in_traffic             | 2020-02-07 14:57:05.871000 |     3715865.43449           |       14863.4617379         |           0.0               |      205071.888889          |
+-------------------------------------+----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+

What is changed and how it works?

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Signed-off-by: crazycs <[email protected]>

crazycs520 · 2020-02-07T13:34:07Z

/run-unit-test

crazycs520 · 2020-02-07T13:39:21Z

/run-unit-test

crazycs520 · 2020-02-07T13:44:35Z

/run-unit-test

crazycs520 · 2020-02-07T13:53:02Z

/run-unit-test

crazycs520 · 2020-02-07T13:56:49Z

/run-unit-test

crazycs520 · 2020-02-11T03:10:57Z

/rebuild

lonng

Some metric table does not contain a comment, is that expected?

crazycs520 · 2020-02-11T04:02:32Z

@lonng, Some metric table name is easy to understand, so they don't have a comment.

infoschema/metric_table_def.go

Deardrops · 2020-02-11T04:14:54Z

infoschema/metric_table_def.go

+	"node_network_interface_speed": {
+		PromQL:  `node_network_transmit_queue_length{$LABEL_CONDITIONS}`,
+		Labels:  []string{"device", "instance"},
+		Comment: "node_network_transmit_queue_length = transmit_queue_length value of /sys/class/net/<iface>.",


Are node_network_interface_speed and node_network_transmit_queue_length same thing?

The comment was from tidb-ansible.
https://github.com/pingcap/tidb-ansible/blob/master/scripts/node.json#L7477

Co-Authored-By: Maxwell <[email protected]>

lonng

LGTM

Deardrops

LGTM

crazycs520 · 2020-02-11T06:19:04Z

/run-all-tests

Signed-off-by: crazycs <[email protected]>

Deardrops

LGTM

iinfoschema/metric_schema: add node cpu/mem/net load metric tables

bc5cb06

crazycs520 added the type/usability label Feb 7, 2020

crazycs520 added 2 commits February 7, 2020 15:32

add comment

f91d1f5

Signed-off-by: crazycs <[email protected]>

Merge branch 'master' into metric-node

7eb96d6

crazycs520 added 2 commits February 10, 2020 11:55

Merge branch 'master' into metric-node

5151c78

Merge branch 'master' into metric-node

4b72b5e

crazycs520 changed the title ~~iinfoschema/metric_schema: add node cpu/mem/net load metric tables~~ infoschema/metric_schema: add node cpu/mem/net load metric tables Feb 10, 2020

Merge branch 'master' into metric-node

0c8042f

lonng reviewed Feb 11, 2020

View reviewed changes

Deardrops reviewed Feb 11, 2020

View reviewed changes

crazycs520 and others added 5 commits February 11, 2020 13:11

Update infoschema/metric_table_def.go

f87124b

Co-Authored-By: Maxwell <[email protected]>

Update infoschema/metric_table_def.go

618c7dc

Co-Authored-By: Maxwell <[email protected]>

Update infoschema/metric_table_def.go

bb6d00c

Co-Authored-By: Maxwell <[email protected]>

Update infoschema/metric_table_def.go

ac32ac1

Co-Authored-By: Maxwell <[email protected]>

Merge branch 'master' into metric-node

00285d2

lonng reviewed Feb 11, 2020

View reviewed changes

Deardrops previously approved these changes Feb 11, 2020

View reviewed changes

Merge branch 'master' into metric-node

f34a7ef

fix name

8ec7407

Signed-off-by: crazycs <[email protected]>

crazycs520 dismissed Deardrops’s stale review via 8ec7407 February 11, 2020 06:23

crazycs520 added status/all tests passed status/LGT2 Indicates that a PR has LGTM 2. labels Feb 11, 2020

Deardrops approved these changes Feb 11, 2020

View reviewed changes

crazycs520 merged commit 931b477 into pingcap:master Feb 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infoschema/metric_schema: add node cpu/mem/net load metric tables #14670

infoschema/metric_schema: add node cpu/mem/net load metric tables #14670

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 11, 2020

lonng left a comment

crazycs520 commented Feb 11, 2020

Deardrops Feb 11, 2020

crazycs520 Feb 11, 2020

lonng left a comment

Deardrops left a comment

crazycs520 commented Feb 11, 2020

Deardrops left a comment

infoschema/metric_schema: add node cpu/mem/net load metric tables #14670

infoschema/metric_schema: add node cpu/mem/net load metric tables #14670

Conversation

crazycs520 commented Feb 7, 2020

What problem does this PR solve?

What is changed and how it works?

Check List

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 7, 2020

crazycs520 commented Feb 11, 2020

lonng left a comment

Choose a reason for hiding this comment

crazycs520 commented Feb 11, 2020

Deardrops Feb 11, 2020

Choose a reason for hiding this comment

crazycs520 Feb 11, 2020

Choose a reason for hiding this comment

lonng left a comment

Choose a reason for hiding this comment

Deardrops left a comment

Choose a reason for hiding this comment

crazycs520 commented Feb 11, 2020

Deardrops left a comment

Choose a reason for hiding this comment