Add filter and grok configuration to documentation

[aj-devops-wi]

Has there been any thought to adding filter and grok documentation to this plugin, to turn ELK into more of a turn-key product for Azure integration? Event Hub is a great integration point for getting log data out of Azure into Elastic, but by default much of the diagnostics log data is retrieved in a format that needs heavy filtering and grok-ing.

By comparison, the Azure Blob plugin has plenty of filter and grok examples for getting the data into a usable form, usually only needing minor tweaks: https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob

[jakelandis]

There is a Logstash module (experimental) that provides a turn-key integration for activity logs and DB diagnostics : https://www.elastic.co/guide/en/logstash/current/azure-module.html

I understand that is only scraping the surface of Event Hub integration's. However, without real world usage of many of the data models it is pretty hard to provide recipes like the document you reference.

I would be open to PR(s) with some common configurations for different Azure services (under a top level directory named "recipes" (or the like)).

[aj-devops-wi]

Similar data to what that module provides would be great! Presumably Microsoft and Elastic collaborated on the Azure Module that got the existing SQL and Activity logs in there, hopefully they'll continue the collaboration to other Azure Monitor log types. The SQL data alone is difficult to parse without understanding the underlying model (due to nested JSON objects, etc.).

I'm not an ELK expert - would you just want sample logstash filters in the recipes, or are these generally created as actual ruby filters that would be added to the plugin?

[jakelandis]

would you just want sample logstash filters in the recipes, or are these generally created as actual ruby filters that would be added to the plugin?

Filters or Ruby (via the ruby filter) doesn't matter much. I would expect the difficult part is understanding the data model well enough to make sense of the data. I am sure the community would benefit from that knowledge however it may be presented.

[EmFl]

This is my current filter for parsing Azure diagnostics and iothub logs. If it can help ?

filter {
  ruby { 
    code =>
    "
      if event.get('records').is_a?(Array)
          event.set('[@metadata][doSplit]', true)
      end
    "
  }
  if [@metadata][doSplit] { split { field => "records" } }
  ruby {
    code =>
    "
      if event.get('records') then
        event.get('records').each do |name, value |
          if name == 'properties'
            if value.is_a?(Hash)
              value.each do |propertyname, propertyvalue |
                event.set(propertyname, propertyvalue)
              end
            else
                event.set(name, value)
            end
          else
            event.set(name, value)
          end
        end
        event.remove('records')
      end
    "
  }
  json {
    source => "properties"
    remove_field => [ "properties" ]
  }
  date {
    match => [
      "time",
      "ISO8601"
    ]
    remove_field => "time"
  }
}

[src386]

Here's a filter example working for activity logs: https://github.com/elastic/logstash/blob/master/x-pack/modules/azure/configuration/logstash/azure.conf.erb

That helped me a lot.

[lukasmrtvy]

@src386 that works only for azure plugin.
Did anyone try to concatenate fields (mutliline logs from AKS) ? https://stackoverflow.com/questions/58434311/logstash-parse-azure-event-hub-logs