include_fields processor does not work with dotted keys

[ruflin]

The include_fields processor does not work in case the key contains a dot. See tests added here for this case.

There are different solutions here:

• Modify GetValue in common.MapStr to also look for keys with dots inside instead of directly walking the tree
• Modifying jolokia fetcher to convert dot keys to objects

The first change would be possible with adding

--- a/libbeat/common/mapstr.go
+++ b/libbeat/common/mapstr.go
@@ -296,6 +296,10 @@ func walkMap(key string, data MapStr, op mapStrOperation) (interface{}, error) {
        var err error
        keyParts := strings.Split(key, ".")

+       if v, exists := data[key]; exists {
+               return v, nil
+       }
+

But that does not fully solve the problem. As in the processor the following code is used, the fields which are added after filtering to the filtered fields, will have a nesting without the docs (see Put method)

	for _, field := range f.Fields {
		v, err := event.GetValue(field)
		if err == nil {
			_, err = filtered.Put(field, v)
		}

		// Ignore ErrKeyNotFound errors
		if err != nil && errors.Cause(err) != common.ErrKeyNotFound {
			errs = append(errs, err.Error())
		}
	}

This Put would also cause other issues as we see in the event below. If it's convert to an object classes and the namespaces classes.loaded would conflict because one is an object and the other one isn't. I would have expected that this already causes an issue on the Elasticsearch side before filtering.

"jolokia":{
   "metrics":{
      "Metrics":{
         "atomikos.nbTransactions":0.000000,
         "classes":18857.000000,
         "classes.loaded":19127.000000,
         "classes.unloaded":270.000000,
         "counter.servo.discoveryclient-httpclient_createnew":13542.000000,
         "counter.servo.discoveryclient-httpclient_delete":13534.000000,
         "uptime":406037194.000000
      },
      "java":{
         "Threading":{
            "DaemonThreadCount":37.000000,
            "ThreadCount":57.000000
         }
      }
   }
}

An alternative solution to the above is replacing the . in the names with _ as we do for example for docker labels and other places with a similar issue. The downside of this is that filtering on fields is not straight forward as the names change between fetching from jolokia and in ES.

See https://discuss.elastic.co/t/include-fields-problem/109809

[andrewkroh]

I would have preferred "modifying jolokia fetcher to convert dot keys to objects" if it weren't for the data being problematic as you have shown.

We could write separate functions to Get/Put that work on keys containing dots. The Get would be more expensive. Get(key string) could try to find a key by successively concatenating keys. Put(key []string, v interface{} would need to explicitly specify the keys.

For example, with Get("jolokia.metrics.Metrics.atomikos.nbTransactions"):

// try map["atomikos"] and if not found then:
// try map["atomikos.nbTransactions"]

[ruflin]

I did some testing on the ES side an the following even with dots will no index in newer versions of Elasticsearch:

         "classes":18857.000000,
         "classes.loaded":19127.000000,
         "classes.unloaded":270.000000,

The reason is that Elasticsearch does the "dedotting" and converstion to an object on its side. That means if someone wants to index the above it is required on our side to replace . with _ or use the trick we do in Metricbeat that it would be classes.value if it is both. But I feel like that adds too much magic.

[ruflin]

Here is a more general issue to discuss this: #5942

[ruflin]

I opened the PR here which dedots the keys and replaces them with _: #5957

[ruflin]

@andrewkroh I'm undecided on what we should do with this PR. The trade off is that the initial document and the created document are not 100% the same as Put is used. But that should be ok.

[ruflin]

Reopening this PR as I think the automatically closing was unintentional.

[andrewkroh]

Related: #2921

[andrewkroh]

Maybe should have a way to escape the dots? Or could offer a way to specify keys similar to Logstash like [metrics][atomikos.nbTransactions]?

[ruflin]

Also looking at #2921 the solution in this PR would work for these cases. The only issue with this solution is that the document looks different before and after the processing. But as from a ES perspective a.b: {2} is the same as a: { b: {2}} anyways, I don't think this is an issue.

If we go with the above change, dot notation can be used in all processors as the walkMap function is changed.

[ruflin]

@andrewkroh Working on #6292 I realised this feature is also needed there. Ok to get this change in?

[ruflin]

I'm closing this in favor of #6406