drop_event does not work in a Metricbeat module filter

[andrewkroh]

• Version: 5.0.0-alpha5
• Operating System: Any
• Steps to Reproduce:

metricbeat:
  modules:
    - module: system
      metricsets: [filesystem]
      period: 15s 
      filters:
        - drop_event:
            when:
              regexp:
                mount_point: '/dev'
output.console:
  pretty: true

When the event is published this is what you get. Notice the null. What should happen is the whole event is dropped.

{
  "@timestamp": "2016-08-11T23:03:34.988Z",
  "beat": {
    "hostname": "myhost",
    "name": "myhost"
  },
  "metricset": {
    "module": "system",
    "name": "filesystem",
    "rtt": 197
  },
  "system": {
    "filesystem": null
  },
  "type": "metricsets"
}

[andrewkroh]

@ruflin Should filters be renamed to processors in Metricbeat? The examples on this page show processors, but it's still filters in the code. https://www.elastic.co/guide/en/beats/metricbeat/current/filtering-and-enhancing-data.html

The examples on that page are wrong. The fields: ['redis.info.memory'] should be fields: ['memory'] due to how/when the filtering is applied. Or do you think we should change when the filtering is applied so that you can write redis.info.memory? Being able to use redis.info.memory is better because if there are two metricsets in the redis module producing a field named memory you can easily choose the correct metricset to filter.

[ruflin]

I remember we had the discussion about filters vs processors in the past. I think the conclusion was to keep it filter here and if possible only expose the pure filter capabilities. Perhaps worth discussing again.

For the fields: As it is inside the module config I would expect fields: ['info.memory']. That would also work with multiple Metricsets.

[andrewkroh]

After thinking about it a bit more, the benefit of having module specific filters is that you don't waste CPU cycles running the filter on all events and you don't need as many conditions on your filters since they only apply to that module.

If we require uses to specify the full field name like fields: ['redis.info.memory'], then we have consistent behavior with the global processors which is easier for us to explain and it's less likely to confuse someone IMO. The cost is that users need to include the module name in the field names (a small cost).

If we decide to go with fields: ['redis.info.memory'] then we should rename the module specific filters to processors for more consistency.

[andrewkroh]

I opened PR #2256 to address the drop_event filtering issue. It does not change any of the naming or how the filters are applied within the module.

[ruflin]

+1 on useing redis.info.memory as I agree it is very nice to have it consistent.

The reason I'm kind of hesitant to use processors is as I'm not sure yet if all future capabilities of processors will be / should be available already on the module level before the final event was generated. This would also mean that the filters are only a subset of processors and not all functionality is available on the module level.

[andrewkroh]

I just realized that if we changed the filters to be applied post event creation that this would change the behavior of include_fields. If you set include_fields: ["redis.info.memory"] then this would end up dropping the metricset field because it's not one of the mandatory fields.

[ruflin]

@andrewkroh Thanks for the fix. We should definitively apply "filters" before the event creation. To still use the full field names we could probably add some magic?