Per subsystem resource usage tracking

[drahnr]

Problem:

When we exhaust resources, we have currently little knowledge about which subsystem consumes how much CPU or memory.

Caveats:

We are fully async on the node side. Hence this cannot be done on a per thread/process level but must be more granular on the logical tasks level.

Hints:

• jemalloc allows setting up multiple pools
• extract cpu usage share from the executor in use
• substrate has some primitives avail iirc, ask @tomaka - if insufficient, consider tokio-console or related

[ordian]

for CPU see Substrate Service Tasks dashboard, and metrics for spawned futures in
https://github.com/paritytech/substrate/blob/b674bd2338b645c41a7f2c21e696891c7e34500d/client/service/src/task_manager/mod.rs

[sandreim]

We need to align on the subsystem/task name formatting. I propose to use snake_case for the subsystem/task name or even use SCREAMING_SNAKE_CASE for the subsystem name. IMO this makes it easier to read and stay consistent with variable, function, method, etc names in the codebase.

WDYT ? @drahnr @ordian @bkchr

[drahnr]

snake_case sounds good to me.

[ordian]

I don't have a strong opinion as long as it's consistent :)

[sandreim]

Discussed with @bkchr and he suggested we go for kebab-case as this is already the naming convention in use for most of the task names. So, it sounds reasonable to me and requires far less changes. Anyone have a strong opinion against kebab-case ? @drahnr @ordian

[sandreim]

Per task memory usage turned out to be more complicated (#4299). Closing this for now ...