Improve Cluster Config Sync #6716

dnsmichi · 2018-10-23T11:36:24Z

Improve Cluster Config Sync

Purpose of this document is to provide solutions for existing sync problems and broken stages.

Scenarios

Client doesn't have (global) zone configured where objects reference to

The master configuration validates fine with a combination of a global zone for templates and the satellite host configuration.

vim zones.d/global-templates/templates.conf

template Host "linux-host-tmpl" {
  check_command = "hostalive"
}

vim zones.d/satellite/hosts.conf

object Host "linux-sat1" {
  import "linux-host-tmpl"
}

The master's zones.conf looks like this:

vim zones.conf

object Zone "master" { ... }
object Zone "satellite" { ... }
object Zone "global-templates" { global = true }

The satellite as client instance doesn't have global-templates configured yet, and will deny the synced configuration.

vim zones.conf

object Zone "master" { ... }
object Zone "satellite" { ... }

This leads to an incomplete configuration as the host object wants to import a template which does not exist.

[2018-09-28 12:06:05 +0200] critical/config: Error: Import references unknown template: 'linux-host-tmpl'
Location: in icinga2b/lib/icinga2/api/zones/satellite/_etc/hosts.conf: 4:3-4:26
icinga2b/lib/icinga2/api/zones/satellite/_etc/hosts.conf(2):   import "linux-host-tmpl"
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^
icinga2b/lib/icinga2/api/zones/satellite/_etc/hosts.conf(3): }
icinga2b/lib/icinga2/api/zones/satellite/_etc/hosts.conf(4):

[2018-09-28 12:06:05 +0200] critical/config: 1 error

Solution

Do not immediately sync the received configuration into /var/lib/icinga2/api/zones but use a staging directory first.
Validate the staged configuration first, and when successful, copy it into production.

Client receives configuration via config::Update message
Stage directory is purged, config diff is taken between production and cluster message config
Configuration is written to the stage directory
All paths for this sync are collected for later processing
Only the current synced files are taken into account for a change - no extra configuration will be copied (deleted files, etc.)
When a config change is detected, this spawns a new process which runs the config validation
A callback is registered upon completion for:
Exit-Code 0: Validation was ok, rmdir(api/zones) and cp(api/zones-stage api/zones). This then triggers the full restart.
Error: Configuration restart aborted, nothing is copied from the stage to production. The error output is logged into the ApiListener status for REST API access.

This solves

#4354
#4301

User renames/deletes zone configuration in zones.conf on the client

This isn't related to the config sync per se, and requires configuration changes followed by reloads on the client itself (admin action).

Currently, the entire configuration in /var/lib/icinga2/api/zones is included during config compilation.
At this stage in the config compiler, we don't have any objects activated yet, so we cannot check against non-configured zone objects unfortunately.

/*
object Zone "global-templates" { global = true }
*/

This leads into this perpetum mobile situation:

The configuration is loaded and compiled ...

mbmif /usr/local/tests/icinga2/master-slave (master *+) # cat icinga2b/lib/icinga2/api/zones/global-templates/_etc/commands.conf
object CheckCommand "sleep" {
  command = [ "/bin/sleep", 30 ]
}

... but the config validation fails since the Zone object 'global-templates' does not exist.

[2018-09-28 11:45:47 +0200] critical/config: Error: Validation failed for object 'sleep' of type 'CheckCommand'; Attribute 'zone': Object 'global-templates' of type 'Zone' does not exist.
Location: in icinga2b/lib/icinga2/api/zones/global-templates/_etc/commands.conf: 1:0-1:26
icinga2b/lib/icinga2/api/zones/global-templates/_etc/commands.conf(1): object CheckCommand "sleep" {
                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
icinga2b/lib/icinga2/api/zones/global-templates/_etc/commands.conf(2):   command = [ "/bin/sleep", 30 ]
icinga2b/lib/icinga2/api/zones/global-templates/_etc/commands.conf(3): }

[2018-09-28 11:45:47 +0200] critical/config: 1 error

Solution

Really a problem, since we can only guess which zones may survive the config compiler. Since we would end in a broken configuration validation later on if zones contain errors, we can use this advantage to enforce a guess.

How?

By making the bold statement that you cannot sync zones via cluster zone sync for this specific instance.

This way, we can fairly assume that the configuration for zones was done statically/via API package.
The solution is to check whether a configitem of the type "Zone" and the directory name already exists.

/* We don't have an activated zone object yet. We may forcefully guess from configitems
 * to not include this specific synced zones directory.
 */
if(!ConfigItem::GetByTypeAndName(Type::GetByName("Zone"), zoneName)) {
        Log(LogWarning, "config")
                << "Ignoring directory '" << path << "' for unknown zone '" << zoneName << "'.";
        return;
}

This solves #3323

Upon next config sync, the directories are automatically purged then (requires the solution with staged syncs).

User renames zones on the client

If the production code imports a template which was previously available in a zone, this is a different error.
The user forcefully renamed something and broke the production configuration.

Future versions of Icinga 2 won't trigger this problem this visible since the cluster config sync is now staged, and production config is synced from above, if valid.

If the user renames a zone, and doesn't copy the zones directory in /var/lib/icinga2/api/zones, manual actions will still be needed. Icinga 2 may detect a difference, but doesn't know which zone was renamed from A to B.

If such zone renames take place, the best option still is to purge /var/lib/icinga2/api/zones manually and restart the client.

Since zone renaming usually happens in the early stages of a cluster design, this problem is not so important than with syncing partial configuration or including removed zones.

User renames zone on the config master or deletes files

Simulate an empty directory where now all files are removed upon sync.

cd icinga2a/etc/icinga2/zones.d/
mv master master-test
mkdir master

The config sync receives an empty directory then, and needs to diff that. This leads into possible errors
and the files are left intact in production.

Solution

With the cluster config sync, we need to purge staging before putting any new files in there. Since the config::Update message doesn't contain any reference, the diff mentioned before takes place. The defined state
from the cluster message is put into staging and then put into production, without any leftovers.

This requires the cluster config sync stages with additional cleanup handling.

This solves #4191

Cleanup

Stage Cleanup

On a new sync via cluster message, the stage directory is purged entirely.

This is mainly to ensure that deleted configuration files and directories (on the config master) are not left there, and would accidentally cause false positives on config validation.

This also forbids custom modifications made by the local user ...

mbmif /usr/local/tests/icinga2/master-slave (master *+) # vim icinga2b/lib/icinga2/api/zones-stage/master/_etc/myown.conf

<restart here>

mbmif /usr/local/tests/icinga2/master-slave (master *+) # ls icinga2b/lib/icinga2/api/zones-stage/master/_etc/myown.conf
ls: icinga2b/lib/icinga2/api/zones-stage/master/_etc/myown.conf: No such file or directory

... for example when attempting to fix the synced configuration. This really needs to be fixed on the config master.

Production Cleanup

When the stage configuration validation went fine, the production configuration is purged away.

This is to ensure that deleted or modified files are really deleted in production, and the same
config files are loaded into memory as validated before inside the stage.

Up until the successful validation, the broken stage validation remains "active" and users can read the "startup.log" and "status" files generated on disk.

Notes

ref/NC/581835

The text was updated successfully, but these errors were encountered:

dnsmichi · 2018-10-25T09:43:16Z

This also includes improved logging for configuration with the origin from endpoint where we keep an authoritative copy (the central config master). This isn't a warning either, so its information now.

dnsmichi · 2018-10-25T09:48:08Z

Design Drafts

From what I did last summer .. just joking, a few weeks ago.

dnsmichi · 2018-10-25T09:49:55Z

Sync local zones directory

dnsmichi · 2018-10-26T12:15:20Z

@lippserd Here's a first draft for local config checksums between received config and local production configuration.

No change

Changes

dnsmichi · 2019-04-15T14:48:00Z

Another round of explanations with @Al2Klimov

My task is still to finalize the PR implementation.

dnsmichi · 2019-04-15T14:50:07Z

I'm moving the part with "User renames/deletes zone configuration in zones.conf on the client" into a separate PR.

The culprit is that we're in compiling configuration stage here, we don't have access to `Zone::GetByName()` as objects have not been activated yet. Our best guess is from a config item loaded before (e.g. from zones.conf) since no-one can sync zones via cluster config sync either. It may not be 100% correct since the zone object itself may be invalid. Still, if the zone object validator fails later, the config breaks either way. The problem with removal of these directories is dealt by the cluster config sync with stages. refs #6727 refs #6716

dnsmichi · 2019-04-25T13:25:40Z

ref/NC/509507
ref/NC/469941
ref/NC/581835
ref/NC/481777

dnsmichi · 2019-06-06T15:44:33Z

Sync child object with missing template in master zone

michi@mbpmif ~/dev/testing/i2-local-cluster (master *=) $ mkdir etc-a/icinga2/zones.d/agent

michi@mbpmif ~/dev/testing/i2-local-cluster (master *=) $ cat etc-a/icinga2/zones.d/master/agent-tmpl.conf
template Host "agent-host" {
  check_command = "random"
  vars.bumsti = "keksi"
}

michi@mbpmif ~/dev/testing/i2-local-cluster (master *=) $ mkdir etc-a/icinga2/zones.d/agent

michi@mbpmif ~/dev/testing/i2-local-cluster (master *=) $ cat etc-a/icinga2/zones.d/agent/host.conf
object Host "sync-agent" {
  import "agent-host"
}

Master 2

Receives the configuration, it is valid (since the template exists in the master zone). Reload is triggered.

[2019-06-06 16:56:52 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
[2019-06-06 16:56:52 +0200] critical/ApiListener: Comparing old (0): '{}' to new (4): '{"/.authoritative":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","/.checksums":"7ede1276a9a32019c1412a52779804a976e163943e268ec4066e6b6ec4d15d73","/.timestamp":"c5ab56a3b481ce564a3a46e820c27735576394a1d0e2121523e06c2d7bd82e7b","/_etc/host.conf":"35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44"}'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/agent//.checksums' for zone 'agent'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/agent//.timestamp' for zone 'agent'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Applying configuration file update for path 'var-b/lib/icinga2/api/zones-stage/agent' (154 Bytes).
[2019-06-06 16:56:52 +0200] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the checksums.
[2019-06-06 16:56:52 +0200] critical/ApiListener: Comparing old (0): '{}' to new (4): '{"/.authoritative":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","/.checksums":"eb5b7d42d1e56e5ff11534eb41dcce72bf459b873274cd7976868ccc9946046a","/.timestamp":"89c41ee9b7b80f6d442b5bee86e9b6c85fa6c01ac06c97e27f615a12a1e60e0a","/_etc/agent-tmpl.conf":"ea8a1dbf48e3967f05300fd885303955f1209e25634aabeb8b636d5ebe4a7d24"}'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/master//.checksums' for zone 'master'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/master//.timestamp' for zone 'master'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Stage: Updating received configuration file 'var-b/lib/icinga2/api/zones-stage/master//_etc/agent-tmpl.conf' for zone 'master'.
[2019-06-06 16:56:52 +0200] information/ApiListener: Applying configuration file update for path 'var-b/lib/icinga2/api/zones-stage/master' (191 Bytes).
[2019-06-06 16:56:52 +0200] information/ApiListener: Received configuration from endpoint 'master1' is different to production, triggering validation and reload.
[2019-06-06 16:56:52 +0200] information/ApiListener: Config validation for stage 'var-b/lib/icinga2/api/zones-stage/' was OK, replacing into 'var-b/lib/icinga2/api/zones/' and triggering reload.

Agent

Config is received but the stage is broken.

[2019-06-06 16:56:51 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master1'. Comparing the checksums.
[2019-06-06 16:56:51 +0200] critical/ApiListener: Comparing old (0): '{}' to new (4): '{"/.authoritative":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","/.checksums":"7ede1276a9a32019c1412a52779804a976e163943e268ec4066e6b6ec4d15d73","/.timestamp":"c5ab56a3b481ce564a3a46e820c27735576394a1d0e2121523e06c2d7bd82e7b","/_etc/host.conf":"35d4823684d83a5ab0ca853c9a3aa8e592adfca66210762cdf2e54339ccf0a44"}'.
[2019-06-06 16:56:51 +0200] information/ApiListener: Stage: Updating received configuration file 'var-c/lib/icinga2/api/zones-stage/agent//.checksums' for zone 'agent'.
[2019-06-06 16:56:51 +0200] information/ApiListener: Stage: Updating received configuration file 'var-c/lib/icinga2/api/zones-stage/agent//.timestamp' for zone 'agent'.
[2019-06-06 16:56:51 +0200] information/ApiListener: Stage: Updating received configuration file 'var-c/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'.
[2019-06-06 16:56:51 +0200] information/ApiListener: Applying configuration file update for path 'var-c/lib/icinga2/api/zones-stage/agent' (154 Bytes).
[2019-06-06 16:56:51 +0200] information/ApiListener: Received configuration from endpoint 'master1' is different to production, triggering validation and reload.
[2019-06-06 16:56:52 +0200] critical/ApiListener: Config validation failed for staged cluster config sync in 'var-c/lib/icinga2/api/zones-stage/'. Aborting. Logs: 'var-c/lib/icinga2/api/zones-stage//startup.log'

It even points us to a log file ...

dnsmichi · 2019-06-06T15:45:03Z

I've just found a small issue with syncing that .authoritative marker.

dnsmichi · 2019-06-19T15:05:50Z

As always, technical concept plus upgrading docs are provided.

dnsmichi added enhancement New feature or request area/distributed Distributed monitoring (master, satellites, clients) labels Oct 23, 2018

dnsmichi added this to the 2.11.0 milestone Oct 23, 2018

dnsmichi self-assigned this Oct 23, 2018

dnsmichi mentioned this issue Oct 24, 2018

Improve cluster config sync #6727

Merged

dnsmichi mentioned this issue Feb 22, 2019

Check validity of endpoint configuration through zone-api #6962

Closed

dnsmichi mentioned this issue Mar 12, 2019

[dev.icinga.com #11740] .conf file will remain in apifolder if the zonesfolder in zones.d is removed or the last .conf-file is removed from /zones.d/zonefolder. #4191

Closed

dnsmichi assigned Al2Klimov Mar 18, 2019

dnsmichi pinned this issue Mar 24, 2019

dnsmichi mentioned this issue Apr 15, 2019

Ignore synced config zones where no config item exists #7120

Merged

dnsmichi mentioned this issue Apr 25, 2019

Replace self-written filesystem ops with boost.filesystem #7102

Merged

dnsmichi added the ref/NC label Apr 25, 2019

dnsmichi mentioned this issue Apr 25, 2019

Fix problem when adding endpoint through API #6473

Open

dnsmichi added the blocker Blocks a release or needs immediate attention label Apr 29, 2019

dnsmichi added the queue/wishlist label May 9, 2019

dnsmichi mentioned this issue May 10, 2019

Automatically sync all data to new nodes in an existing zone #7176

Open

lxf1992521 mentioned this issue May 23, 2019

Failed to delete "old active stage" when "new active stage" have created, icinga api return 500 status. #7194

Closed

dnsmichi mentioned this issue Jun 6, 2019

[dev.icinga.com #12119] Use config staging for cluster config sync #4354

Closed

dnsmichi closed this as completed in #6727 Jun 19, 2019

dnsmichi mentioned this issue Jun 27, 2019

Icinga2 Agent stuck in an infinite config sync loop #7258

Closed

dnsmichi unpinned this issue Jul 30, 2019

This was referenced Sep 20, 2019

2.11.0 Problems with CheckCommand synchronization through global-zone #7516

Closed

Problem with zones after upgrading to 2.11 #7519

Closed

dnsmichi mentioned this issue Oct 8, 2019

Config sync checksum change detection may not work within high load clusters #7565

Closed

2 tasks

dgoetz mentioned this issue Jan 14, 2022

If a zone managed via the API endpoint package is emptied, it still stays in the synchronized directory on second master #9175

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Cluster Config Sync #6716

Improve Cluster Config Sync #6716

dnsmichi commented Oct 23, 2018 •

edited

Loading

dnsmichi commented Oct 25, 2018

dnsmichi commented Oct 25, 2018

dnsmichi commented Oct 25, 2018

dnsmichi commented Oct 26, 2018

dnsmichi commented Apr 15, 2019

dnsmichi commented Apr 15, 2019

dnsmichi commented Apr 25, 2019

dnsmichi commented Jun 6, 2019

dnsmichi commented Jun 6, 2019

dnsmichi commented Jun 19, 2019

Improve Cluster Config Sync #6716

Improve Cluster Config Sync #6716

Comments

dnsmichi commented Oct 23, 2018 • edited Loading

Improve Cluster Config Sync

Scenarios

Client doesn't have (global) zone configured where objects reference to

Solution

User renames/deletes zone configuration in zones.conf on the client

Solution

User renames zones on the client

User renames zone on the config master or deletes files

Solution

Cleanup

Stage Cleanup

Production Cleanup

Notes

dnsmichi commented Oct 25, 2018

dnsmichi commented Oct 25, 2018

Design Drafts

dnsmichi commented Oct 25, 2018

Sync local zones directory

dnsmichi commented Oct 26, 2018

No change

Changes

dnsmichi commented Apr 15, 2019

dnsmichi commented Apr 15, 2019

dnsmichi commented Apr 25, 2019

dnsmichi commented Jun 6, 2019

Sync child object with missing template in master zone

Master 2

Agent

dnsmichi commented Jun 6, 2019

dnsmichi commented Jun 19, 2019

dnsmichi commented Oct 23, 2018 •

edited

Loading