Object triggered verbose logging

[kenmarsh8]

Is your feature request related to a problem? Please describe.

In trying to debug configurations, and possibly Icinga2, it is common to advise that debug mode be turned on. However our system logs 5 to 10MB a minute, so it is impractical to leave it on overnight. I can turn it on and off, but it is difficult to catch the events in question, and to wade through all the logging not relevant to my issue.

Describe the solution you'd like

I would like to flag a particular object (e.g. Host, Notification, Service) and log just interactions with that object, including the chain of other objects. For example if I flag a Host object, if it is involved in a parent dependency, log that, if it has a Service that goes soft and hard, log that, and that service notifies, log that.

If I flag a Service, log soft and hard state and notifications or invoking a dependency, but not everything else that involves the parent Host.

Describe alternatives you've considered

There's turning it on and off as mentioned above.

Additional context

Add any other context or screenshots about the feature request here.

[Al2Klimov]

Implementation idea:

• Log instances take (maybe via template(?) varargs, maybe even via <<, TBD) also const ConfigObject::Ptr& immediately involved in the logged event
• ConfigObject provides a virtual method which returns immediately related objects (if our existing dependency graph is not flexible enough, TBD)
• The loggers collect the config objects, filter them based on a DSL function on the logger (or flag on ConfigObject, TBD) and decide then

[Al2Klimov]

@julianbrost What about this one for e.g. 2.15? Just thought about this during work on a ticket from a customer with a probably large environment about a single checkable. I guess this feature would help us with debugging in the first place.

[julianbrost]

I mean if you're building something that's useful and reusable anyways, make it a PR and we can look at it.

But the proposal in #7912 (comment) sounds like an invitation for performance issues. Graph lookups and DSL function calls for every potential debug log message sounds really expensive.

[Al2Klimov]

I mean if you're building something that's useful and reusable anyways, make it a PR and we can look at it.

I don't have anything in stash, don't worry. But the earlier we put this on rails, the earlier (or probably not) customers won't have to worry about excessive debug log 🪵 I/O just for providing us some info about object X.

But the proposal in #7912 (comment) sounds like an invitation for performance issues. Graph lookups and DSL function calls for every potential debug log message sounds really expensive.

OK, forget about the DSL. Let's just imagine every object has a log level attribute. Every message of this level (e.g. notice) or higher (info, warn, ...) affecting the object (in)directly is logged despite logger's level. Actually the "graph" is just a big parent<->child map. It can be shared-locked if not already done. And the log levels in question can be cached like: Oh, a debug message, but there are 0 objects with that level, so nothing to look up in the graph anyway. Apropos lookup, this would be like: For each supplied involved object (usually one or two) get all parents (e.g. for notification: its service, its host, its host group), get their log levels, compute the most verbose one.

Does this sound like a plan?

[kenmarsh8]

I mean if you're building something that's useful and reusable anyways, make it a PR and we can look at it.

But the proposal in #7912 (comment) sounds like an invitation for performance issues. Graph lookups and DSL function calls for every potential debug log message sounds really expensive.

Extreme write pressure to the storage system is another kind of expensive.

[Al2Klimov]

@julianbrost Also in 2.14 we've forbidden runtime updates to relationship attributes. This enables caching of all parents per object according to the graph from object creation during its whole lifetime.

[Al2Klimov]

@julianbrost In #9844 you've put up the question about the destination of such privileged log messages. 💡 What about instead (of that whole concept) letting every logger optionally define a set of involved object types and names of objects to limit itself to?

E.g. on our behalf a customer API-creates/deletes (ciao reload!) a FileLogger with debug level, but a specification like "Host" => "dns1.netways.de". That logger will log only stuff related* to dns1.netways.de.

* Implementation detail

[julianbrost]

You mean something like that?

object FileLogger "debug-file" {
  severity = "debug"
  path = "/var/log/icinga2/debug.log"
  object_filter = {
    "Host" = ["mail.example.org"]
  }
}

So if there was no such logger with severity = "debug", the existing checks suffice to filter out debug messages, so no new overhead in that case. If there was an unfiltered debug logger, things would pretty much be as they are right now as well. Only if there was a filtered logger, there would be some extra work within that logger. Sounds like a plan, the next question will be how that filtering will look like.