Ability to add custom fields to all beats

[msimos]

Right now you can only add tags with topbeat. But it would be useful to add custom fields like "myfield": "foobar". Filebeat can do this using the prospectors & fields directive.

[jamesmr89]

I'd like to add a +1 to this

[ruflin]

Would you expect these fields to be set on a global beat level or per document type?

[jamesmr89]

I think for the case I'm targeting global would be desired.

I run some 120 second tests on a device and would like to capture top beat status during that time, and add a field of something like TestID: 123456 so that I could later see the results for this specific test.

Does that make sense?

If I understand the question correctly, If it were global, in theory i'd be able to do the same thing for packetbeat as well?

[ruflin]

@jamesmr89 Correct. The reason I was asking is to better understand the need. In filebeat it is local to each prospector. Could your case also be covered by the global tags?

[jamesmr89]

Yea, I think that would work for me, I use it like described in filebeat I think the same type of config is exactly what I'm looking for.

[ruflin]

I was referring to the existing tags feature above: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#_tags Does that work for you?

[jamesmr89]

Ok see what you are saying. It doesn't exactly work for me. I do a test run which may have many tests in it so there is both a testid and a runid in my case so with the tag I can't really determined between the two sometimes this is why I think I need a field. The field seems to give flexibility that would probably fit other use cases I'm not considering

[ruflin]

Ok, good to know.

[monicasarbu]

We could add in Packetbeat and Topbeat the option to configure custom fields as we do for Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#configuration-fields. The fields can be global in Topbeat and for Packetbeat can be configured per protocol.

[monicasarbu]

We could add in Packetbeat and Topbeat the option to configure custom fields as we do for Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#configuration-fields. The fields can be global in Topbeat and for Packetbeat can be configured per protocol.

[jamesmr89]

Monica, That would make my day... Thanks

[ruflin]

As usual when we talk about introducing new fields, I worry about conflicts :-) Here some suggestion on how we could proceed:

Global Fields

To make fields available to all beats (also community beats), I think it is a good idea to introduce it globally. The problem with introducing it globally on a root level, we would already have now a conflict with filebeat. To prevent this I would suggest to introduce it under the beat namespace: beat.fields I think it belongs under the beat namespace, as they are global and beat specific. Initially I thought about merging fields, but that only calls for troubles if for example the same key exists twice.

Local Fields

I also like the idea of introducing local fields, like we have for prospectors or as monica suggested, per protocol. We should generalise the implementation we have done for filebeat so it can be implemented in other beats easily. That could probably done directly when implementing global fields, as they should depend on the same code base.

Priorities

I would suggest to go with the global fields first as they will directly benefit all beats and so far I think no feature request for "local" fields in other beats popped up.

Naming

We should discuss if "fields" is the best name for this feature or if there are other / better options.

[tsg]

I actually like the idea of merging "global" with "local", making sure that "local" takes precedence in case of conflicts. That wouldn't be hard to do, because we know that "global" will be applied last (in the publisher phase).

For naming, perhaps we could use "labels"? It's more clear from my POV.

[monicasarbu]

I like more labels as it's more descriptive. I agree with Nicolas that we should start with global field. The user can configure labels in the shipper section, near tags:

labels:
  TestID: 123456

and the publisher from libbeat will add the extra fields for all the beats.

[ruflin]

I'm not a fan of merging as this can lead to unexpected behaviour, but at the same time I could even see it done on purpose. A general field is defined and it is overwritten locally. Perhaps we can add a field "conflict: true" if a conflict is found. I think LS does something similar.

For me "labels" are close to tags and normally not key - value pairs. See for example labels on Github.

As for the placement I agree that it that it should be at the same place as tags, but both should be under "beat" and not "shipper" ;-) But this is a separate discussion.

[palmerabollo]

This feature (global fields) would be extremely useful to me, too. I currently have to configure topbeat to log to a file and then use filebeat to read that file and add my own fields (I also use fields_under_root: true).

[lakshmanantokbox]

This features would be very helpful if it would allow us to add a new field like filebeat.

[jmendiara]

It would be great to have this feature, and reach consistency with filebeat

[lentregu]

I also need this feature, please!!!

[Another-Coding-Nerd]

+1 for this feature

[andrewkroh]

This feature has been implemented. See #1092 for usage examples. This will be released in v5. It can be tested now from the 5.0 development builds.