Categorise log errors/warnings coming from Qlik Sense

[mountaindude]

Current situation

The log appenders in Qlik Sense use filters to determine what log events that should be forwarded to Butler SOS.
It can however be difficult to create exactly the needed filters, with the result that filters have to be created a bit too wide/open/accepting. The effect is that some events that are of no (or little) interest may be sent to Butler SOS.

Another scenario is that large number of log events are sent to Butler SOS, making it difficult to understand what they represent.
If Butler SOS could categorise the events they would provide more valuable information and insights.

Suggested change

Add a categorisation mechanism for log events coming into Butler SOS.

• Possible to enable/disable the feature
• Allow for zero or more "rules"
• A rule defines the criteria for a certain log event to be associated with one or more categories
• Categories are represented in InfluxDB by tags on the log event datapoint
• Have a default rule that capture all log events that did not match any other rule
• Different actions for each matched log entry (categorise, drop)

Config file

Butler-SOS:
  ...
  logEvents:
    categorise:                        # Take actions on log events based on their content
      enable: true
      rules:                         # Rules are used to match log events to filters
        - description: Find access denied errors
          logLevel:                    # Log events of this Log level will be matched. WARN, ERROR. Case insensitive.
            - ERROR
          action: categorise            # Action to take on matched log events. Possible values are categorise, drop
          category:                     # Category to assign to matched log events. Name/value pairs. 
                                        # Will be added to InfluxDB datapoints as tags.
            - name: qs_log_category_1
              value: access-denied
            - name: qs_log_category_subsystem
              value: repository
          filter:                       # Filter used to match log events. Case sensitive.
            - type: sw                  # Type of filter. sw = starts with, ew = ends with, so = contains
              value: "Access was denied for User:"
            - type: so
              value: was denied for User
        - description: Find AD issues
          logLevel:                    # Log events of this Log level will be matched. WARN, ERROR. Case insensitive.
            - ERROR
            - WARN
          action: categorise            # Action to take on matched log events. Possible values are categorise, drop
          category:                     # Category to assign to matched log events. Name/value pairs. 
                                        # Will be added to InfluxDB datapoints as tags.
            - name: qs_log_category_1
              value: user-directory
            - name: qs_log_category_subsystem
              value: repository
          filter:                       # Filter used to match log events. Case sensitive.
            - type: sw                  # Type of filter. sw = starts with, ew = ends with, so = contains
              value: Duplicate entity with userId
      ruleDefault:                   # Default rule to use if no other rules match the log event
        enable: true
        category:
          - name: qs_log_category_1
            value: default