Icinga2 does not reliably collect all performance data values during reload

[ix-dev]

When we deploy a new configuration using the Director some performance data is lost, but the actual checks are executed (either by the satellites or the core itself).

Expected Behavior

No performance data should be lost during reload of the Icinga2 core.

Current Behavior

Some random performance data is lost when the core reloads its configuration.

Possible Solution

Buffer check results during the reload phase of the Icinga2 core so that no performance data is lost.

Steps to Reproduce (for bugs)

1. Set check_interval for hostalive checks to 5 minutes
2. Enable the perfdatawriter to collect performance data in a file
3. Deploy new configuration using the Director
4. Check if all performance data is written to file

Context

We want to collect performance counters in a reliable way.

Your Environment

Version used: Icinga2: r2.10.2-1, IcingaWeb2: 2.5.0, Director 1.3.1
Operating Systems: Debian 8

[dnsmichi]

• What exactly happens during such a reload? (Logs, etc.)
• How big is your environment and how long does such a reload take? (Output of icinga2 daemon -C)
• When do these check events occur, and what proof can be seen inside the written performance data files (the gap of timestamps for example).

[ix-dev]

• Since the logs contain sensitive information we could not publish the logs here. Instead, we could offer to send them to Netways directly.
• output of icinga2 daemon -C: icinga2_daemon_c.txt
• hostperfdata with gap: host-perfdata.txt
  The check interval of the hostalive check is set to 5 minutes.

[lazyfrosch]

I guess the problem is caused by queue data getting dropped during reload.

[dnsmichi]

Scheduling for 2.11 to not forget that code is in git master already for this.

[ix-dev]

We tested release 2.10.2 with the patches from pull requests #6882 and #6908 but it does not solve the problem yet. We will test again with the current snapshot and provide feedback.

[dnsmichi]

ref/NC/591065