Metrics report

The metrics layer

Seastar has a metrics layer, which allows registering different kinds of metrics and export them in one of a few reporting API (Collectd, Prometheus or RESTFull).

Registering metrics

To reduce the compilation overhead the definitions are split between the header and source files of the measured class.

For this example, assume that class A has some members x we want to meter.

under A header file

#include "core/metrics_registration.hh"

class A {
   seastar::metrics::metric_groups _metrics;
   int x;
   void setup_metrics();
   A() {
      setup_metrics();
   }
}

Under A source file

#include <seastar/core/metrics.hh>

void A::setup_metrics() {
namespace sm = seastar::metrics;
  _metrics.add_group("AGroup", {
       sm::make_gauge("used_bytes", x,      
   sm::description("bytes used by the x parameter"))
  });
}

Picking the right metric type

Data types are explained here. The way Collectd works, counters and derive metrics will be derived, so for things that the derivation is interesting (like with total number of bytes, where the byte rates is what we are looking) use derived. If you care about the value (like messages in a queue) use gauge. When in doubt, use gauge.

Note: derived are signed 64 bit integer, its probably big enough to anything that is countable.

Metric uniqueness

A metric name should be unique per group. You can use labels to differentiate metrics, but only for metrics of the same type.

For example, each metric contains a shard id label, so a per-shared metric is unique.

Checking your metrics

Run a Prometheus server and search for your metrics. Metrics names are prefixed with seastar_ and the group name.

How to move from collect to metrics

Collectd used to be the way to register metrics. With the introduction of the metrics layer, the preferred way for registration is with the metrics layer.

This describes what need to be done to move a code that was meter using the collectd, to use the metrics layer.

By the end of the transition there should be no dependencies to collectd.

Changes in the header file

Add #include "core/metrics_registration.hh" and remove any collect includes.

replace the _collectd_registration member with:

seastar::metrics::metric_groups _metrics;

Add (or rename setup_collectd to setup_metrics)

remove all collectd related code from the header file.

Changes in the source file

add #include <seastar/core/metrics.hh>

In setup_metrics (Which used to be setup_collectd) do the following:

Instead of:

  _collectd_registrations = std::make_unique<scollectd::registrations>(scollectd::registrations({
        scollectd::add_polled_metric(scollectd::type_instance_id("cache"
                , scollectd::per_cpu_plugin_instance
                , "bytes", "used")
                , scollectd::make_typed(scollectd::data_type::GAUGE, [this] { return _region.occupancy().used_space(); })
        )

You would use something like:

namespace sm = seastar::metrics;

_metrics.add_group("cache", {
       sm::make_gauge("bytes", "used", [this] { return _region.occupancy().used_space(); },  sm::description("bytes used by chache"))
});

Things to notice

• A group of metrics is define in an initilization list passed to the add_group method
• You can call add_group multiple times, even with the same group name
• The metrics are created with make_gauge, make_derive, etc'