New Monitoring

New Roles

DeepSea now knows the prometheus and grafana roles and deploys the monitoring stack accordingly. This allows for e.g. multiple prometheus instances for an HA setup.

There is also code to remove Prometheus and Grafana from nodes that do not have the respective roles. If you have a Prometheus/Grafana installation that is managed outside of DeepSea on DeepSea minions, make sure to add rescind-[prometheus|grafana]: default-nop to your pillar, otherwise DeepSea will remove your installation.

Pillar variables

Default below pillar is available to all nodes by default. Either alter the global in /srv/pillar/ceph/stack/global.yml or alter /srv/pillar/ceph/stack/<cluster_name>/minions/<host> if specific minion configs should be altered. Refer to stack pillar doc

monitoring:
  prometheus:
    rule_files: []
    scrape_interval:
      ceph: 10
      node_exporter: 10
      prometheus: 10
      grafana: 10
    relabel_config:
      ceph: ''
      node_exporter: ''
      prometheus: ''
      grafana: ''
    metric_relabel_config:
      ceph: ''
      node_exporter: ''
      prometheus: ''
      grafana: ''
    target_partition:
      ceph: '1/1'
      node_exporter: '1/1'
      prometheus: '1/1'
      grafana: '1/1'

• scrape_interval: change the scrape interval for various scrape target groups (these groups map to exporters that provide data)
• target_partition: When multiple prometheus instances are deployed it can be desired to partition scrape targets and have some instances scrape only part of all exporter instances (currently this is only implemented for node_exporter targets). Say there're two prometheus instances and the available node_exporter targets should be divided between them. Configure the pillar so that on instance sees monitoring:prometheus:target_partition:node_exporter:'1/2' while the other sees monitoring:prometheus:target_partition:node_exporter:'2/2.