Config directory

[vitaly-m]

@brian-brazil I have prepared simple improvement which will able to split complex configuration file to more atomic files. This improvement is related to #85, I need this improvement because it is really too complex sometimes to manage one huge configuration file or to start several instances of snmp_exporter.

[brian-brazil]

Thank you for you PR. As indicated in #85 this is not something I believe should be in the snmp exporter, as this sort of thing is the role of configuration management.

Also since then the generator has been added, which produces a single file. This leaves you only to edit the much shorter and simpler generator.yml. The general idea is to have one snmp.yml per organisation (secret visibility permitting).

The snmp.yml was never intended to be manually managed in the long term.

[vitaly-m]

In my case configuration may be changed at runtime, for example new modules may be added. I want to automate it and think that it is much easier to automate creation of separate file instead of writing the logic of changing the one huge file especially in concurrent environment.

[brian-brazil]

There's nothing stopping you changing the configuration at runtime. Currently the config is read at every request, but that'll change to the usual Prometheus HUP approach at some point.

May I ask why you're expecting such frequent changes and how you plan on managing them? I'd personally expect this to be done by hand maybe once every few weeks in a busy environment.

[RichiH]

One effect I have seen internally is different teams owning different hardware. They tend not to want to touch other team's stuff if they can avoid it at all because especially non-generated config is a pita to work with.

Personally, I have every config in one file and created a short script to splice that into what snmp_exporter sees.

If we play the 5 whys:

• Raw snmp_exporter configs are huge
• Configs are partially maintained by hand, not generated
• Some teams don't use the generator
• They don't use it for everything so can't build familiarity and it's overhead as they need to maintain two paths
• As they require multiple labels (ifIndex, ifName, etc) in different contexts

[vitaly-m]

May I ask why you're expecting such frequent changes and how you plan on managing them? I'd personally expect this to be done by hand maybe once every few weeks in a busy environment.

It can be pretty often procedure, and it will automated within the other software, so the user will be able to provide a configuration file using some UI, after that the configuration will be sent to snmp_exporter. It is supposed that this configuration will not affect already monitored objects and will take effect only for new one, for example some new metrics can be introduced or credentials will be changed.

[brian-brazil]

If you're building that sophisticated a system, then merging the various configs together is not going to be a big problem technically for you.

How many modules are you expecting? I'm wondering if we might need the HUP stuff sooner rather than later.

[vitaly-m]

Several problems:

1. Additional local storage for configuration files, to understand the current status of applied configuration
2. Concurrent modification
3. Complex logic of file diff and merge
4. Complex analysis of applied configuration in case of problems in it (it definitely will happen)
5. Even more complex removal of unnecessary configuration

We are trying to use snmp_exporter for virtual infrastructure it may change frequently.

[SuperQ]

@vitaly-m All of those problems seem like they will be an issue for individual files as well. It sounds like you need a standard template generator that takes input from an authoritative source. Things like Ansilbe, Chef, Puppet, Salt. They all solve the above problems.

[vitaly-m]

Individual files don't have such problems because they are atomic and immutable, at least in our cases.

[brian-brazil]

Any system like this requires a database of some form to store the configuration plus some form of logging, which you need to handle 1), 2) and 4) in addition to things like redundancy and backups.

3 is trivial as modules are independent.

5 Is actually easier with one configuration file, as extra individual files won't tend to get left behind.

These are all fundamental challenges with the type of system you're trying to build, it's out of scope for the snmp_exporter to try and solve configuration/change management problems like these. We aim to provide primitives on which you can build such a system on top of. The only potential problem I see on our side is if you have a lot of modules, that could slow down queries given we currently load the config file on every request.

We are trying to use snmp_exporter for virtual infrastructure it may change frequently.

You may add and remove targets frequently in such an enviroment, what oids you monitor and the types of devices you have tend to change more slowly.

[SuperQ]

If the individual files are "atomic and immutable", then combining them has none of the problematic effects you describe. cat config/*.yml > snmp.yml is equivalent, and can easily be triggered at update time.

[vitaly-m]

@brian-brazil what do you mean?

Prometheus HUP approach

I'm looking at Prometheus configuration file and it looks like it also do not support the configuration of jobs in a separate files.

[brian-brazil]

I was referring to when the configuration file is loaded. In Prometheus the user must request it.

In the snmp_exporter, it currently happens on every single request.

[vitaly-m]

I think that request of configuration update is the correct way of configuration update, because file watching service depends on OS and FS type, updating of configuration for each call can consume much time especially in case of one exporter with hundreds of modules for thousands targets. Maybe I will do a change for it in snmp_exporter.

[SuperQ]

Yes, switching from on-request to HUP would be a greatly appreciated change.

[brian-brazil]

Seems like we're sorted here.