Allow numbers and boolean values for tags.

[simitt]

Store numbers as scaled_floats, scaling_factor 1000000
fixes #828.

depends on elastic/beats#9772

[felixbarny]

What is the maximum number that can be stored with this scaled float format? Are strings, numbers and booleans stored in the same place in ES? What if one span has the tag foo: bar and another foo: 42 (different data types for the same tag)?

[simitt]

Yes, strings, numbers and booleans are stored at the same place in ES. If the same attribute has different datatypes this will throw an exception.

[simitt]

According to ES doc a scaled_float is :

A floating point number that is backed by a long, scaled by a fixed double scaling factor.

where a long is a signed 64-bit integer with a minimum value of -263 and a maximum value of 263-1.

Having a scaled_factor of 1,000,000 would lead to a max value that can be stored with full precision of 2**63/1000000 which equals 9223372036854.

[graphaelli]

jenkins, retest this please

[graphaelli]

We should follow up with some documentation about sticking to a single data type for a given tag.

[simitt]

@felixbarny I'd like to merge this soonish - do you have any concerns regarding the handling?

[felixbarny]

Do users have to possibility to set another mapping type in case they want to set large numbers?

For most use cases, including recording bytes up to 9PB and Unix timestamps in up to millisecond precision the current range is sufficient.

[felixbarny]

If the user can't configure the mapping type I think it would make sense to validate the range in the APM Server so that the error appears in the agent logs as well rather than only in the APM Server and Elasticsearch logs.

But if possible I think users should be able to configure the mapping type, although the default you choose sounds good to me.

[simitt]

There is a way to add fields to the ES template, see discussions around this #1371, which makes use of the experimental feature append_fields (experimental).
This can not be used for tags, as it raises an exception when used for fields that already have a template.

However, one can make use of the setup.template.path and setup.template.overwrite: true config options to point to a custom fields.yml, and add other types here. The custom fields.yml then needs to be manually updated every time the server is updated, to ensure new mappings are added, so all features work as expected.

So to summarize, there is some hackish way to do add custom mapping types.

[simitt]

jenkins, retest this please.

[simitt]

If the user can't configure the mapping type I think it would make sense to validate the range in the APM Server so that the error appears in the agent logs as well rather than only in the APM Server and Elasticsearch logs.

That is not possible with the setup we have at the moment, as the server would need to either handle the whole request syncronously so errors can be returned back to the agents, or the server would need to fetch all indexed tags with their types and check them for every incoming document, which would require some kind of shared cache to work between multiple servers.

The same handling applies if e.g. a mapping explosion or any other issue with storing the data to ES happens.