metrics: total cpu consumption over time does not make sense

[sanderegg]

This metrics should be re-worked, re-thinked.. it seems to show wrong values

[sanderegg]

We shall use the following metrics to get that number correctly:
It needs to be set as a recording rule.
sum(increase(node_cpu_seconds_total{mode!="idle"}[12w]))

[sanderegg]

The recording rule for prometheus was now added in all deployments.
it means we can begin to use it in grafana where it still needs to be added

[sanderegg]

@Surfict , @mrnicegyu11 : for the grafana dashboards, I cannot do it. so let's discuss that together tomorrow.

[elisabettai]

@sanderegg, does the grafana dashboard for CPU "consumption" looks OK to you know?

For reference, last time (May 2022) we reported "5877 CPU hours over the last quarter".

[sanderegg]

@elisabettai well, I would say that it does not look too bad at a nose level ;)
it looks like the increase in CPU hours is more or less stable, which I think is kind of expected right?
now as to the absolute number... well it means we have currently about 10729 cpu hours?

[sanderegg]

@elisabettai shall we keep that one up or close? you do not rely on this anymore right?

[sanderegg]

@elisabettai shall we close this?

[sanderegg]

closing as outdated

[elisabettai]

@JavierGOrdonnez, you might want to follow up on this.

In the metrics repo, recently the CPU usage graph is giving again very high values, e.g. this graph.

The values we normally reported are around 13’000 CPU-hours (at least in this order of magnitude, see this graph).

fyi @sanderegg, this is the query we use in the metrics repo: 'osparc_node_cpu_seconds_total_nonidle_increase_over_nodes_12weeks/3600'