Feature: Add Puppet Classes view

[lcharreau]

Hello,
Recently, in order to better monitor our Puppet nodes, we needed to add a Class view in Puppetboard.

This view lists the Puppet classes that have had at least one of their resources changed (event). It consists of extracting the resource events from the last report of each node and grouping these events by class.
For each class, we display the number of nodes having executed a resource declared in this class, as well as the number of events by status ('failure', 'success' and 'noop').

The main drawback of this view (which is disabled by default) is that it is potentially resource-consuming since it requires parsing the last report of each node and classifying the events by class.
This heavy processing pushed us to add a cache system to store the results but also a scheduler to pre-compute these results so that they are available more quickly to the user.
If you have more than one worker, it is recommended to use a shared backend like Memcached (that's what we do).
To display the class view, we retrieve the pre-computed result (by a scheduled job triggered at regular intervals, every 5mn in our case) from the cache and we parse only the new reports (i.e. those generated after the caching and which are therefore not present in the cache). We take advantage of this to update the cache. For information, we automatically run Puppet agent every 4 hours.

When we click on a class, we display the view by class. This view allows listing all the nodes whose last report has executed (event) at least one resource declared in the class. The status of the node is also displayed as well as the status of the class (which corresponds to a global status of the resource events).

I submit this new feature without knowing if it can interest the Puppet community.

Screenshot (Classes view)

Screenshot (Class view)

docker-compose configuration used to test this new view:

services:
  puppetboard:
    container_name: puppetboard
    hostname: puppetboard
    image: puppboard:4.2.3_xxx
    restart: always
    ports:
    - 127.0.0.1:8082:80
    volumes:
    - "/etc/puppetlabs/puppetdb/ssl:/etc/puppetlabs/puppetdb/ssl:ro"
    extra_hosts:
    - hostname-puppetdb:192.168.10.10
    environment:
    - PUPPETBOARD_WORKERS=5
    - PUPPETDB_HOST=hostname-puppetdb
    - PUPPETDB_PORT=8081
    - PUPPETDB_PROTO=https
    - PUPPETDB_SSL_VERIFY=/etc/puppetlabs/puppetdb/ssl/ca.pem
    - PUPPETDB_KEY=/etc/puppetlabs/puppetdb/ssl/private.pem
    - PUPPETDB_CERT=/etc/puppetlabs/puppetdb/ssl/public.pem
    - ENABLE_CLASS=True
    - SCHEDULER_ENABLED=True
    - CACHE_TYPE=MemcachedCache
    - UNRESPONSIVE_HOURS=16
    networks:
      app_net:
        ipv4_address: 10.10.30.10
  memcached:
    container_name: memcached
    hostname: memcached
    restart: always
    image: memcached:latest
    networks:
      app_net:
        ipv4_address: 10.10.30.11

networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 10.10.30.0/24

[codecov]

Codecov Report

Patch coverage: 36.65% and project coverage change: -8.29 ⚠️

Comparison is base (d519561) 85.55% compared to head (ff2f806) 77.27%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #799      +/-   ##
==========================================
- Coverage   85.55%   77.27%   -8.29%     
==========================================
  Files          19       20       +1     
  Lines        1087     1307     +220     
==========================================
+ Hits          930     1010      +80     
- Misses        157      297     +140     

Impacted Files	Coverage Δ
puppetboard/views/classes.py	21.95% <21.95%> (ø)
puppetboard/core.py	79.31% <60.00%> (-6.74%)	⬇️
puppetboard/app.py	84.28% <100.00%> (+0.95%)	⬆️
puppetboard/default_settings.py	100.00% <100.00%> (ø)
puppetboard/docker_settings.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[gdubicki]

Hi @lcharreau!

Thank you for your contribution. I think that it’s ok to merge it if the added feature is behind a feature flag and if it requires Memcache config to start the app only if this feature flag is enabled. Does it work this way?

[lcharreau]

Hello @gdubicki,

thank you for your support.
Yes by default the "Classes" view is disabled, so no need to have a backend available for the cache (ENABLE_CLASS).

The default cache backend is "SimpleCache" (in memory), which is acceptable when you have only one worker.
However, if you configure more than one worker, it is recommended to use a shared backend like "MemcachedCache".
The type of backend is configured via the CACHE_TYPE variable.

As I explained in my previous post, you can also add a scheduled task to pre-compute the results at regular intervals (SCHEDULER_ENABLED). By default, this is also disabled.

To have the complete detail of the variables related to this functionality, you can refer to the variables defined in the puppetboard/docker_settings.py and puppetboard/default_settings.py files.

[gdubicki]

Released in v4.3.0 a moment ago. 😊