[Index patterns] Store field-specific metadata from mappings #82318
Comments
wylieconlon
added
enhancement
|
Pinging @elastic/kibana-app-arch (Team:AppArch) |
|
@wylieconlon Thanks for bringing this to my attention. Its definitely something we should do. Can you help me figure out the priority? What would be built on top of this and is it roadmapped? |
|
@mattkime I added some links to this issue from the related issues. Now that the decisions is made at the stack level, solutions like Observability are planning on adding metadata in a few places as early as 7.11. The most basic implementation I would hope for is that we would assign a different field formatter when the metadata indicates the type |
exalate-issue-sync
bot
added
impact:low
petrklapka
added
Team:DataDiscovery
|
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery) |
|
@mattkime would be interesting to think about the size/effort, sounds more like hours? |
|
Just talked with @felixbarny and it would be good to have the |
|
so the request would be to apply auto field formatting given the meta.unit fields was available via field_caps request is this correct? FYI @mattkime @davismcphee |
|
Yes, it is correct. |
|
It sounds like a reasonable ask to me 👍 It would be convenient for users to only have to configure this once in ES instead of having to reconfigure it on the Kibana side too. |
|
@dej611 @felixbarny Is there an example data set you're using? Whats the best way for me to find a |
|
This page documents the conventions around In this Elasticsearch PR, I'm proposing to prefer UCUM symbols but it keeps compatibility with the existing conventions. So after that PR has been merged, we should also update Kibana accordingly. |
|
field caps returns an array for How should we handle a different formats are supplied by different indices? aka format conflicts. This PR makes field meta content available - #174973 - more work needs to be done to make it useful for formatters. @felixbarny Can you supply a pair list of unit strings and the cooresponding kibana formatters that need to be applied? |
|
I'm not really familiar with Kibana formatters. Could you please propose a table of the mappings from |
Good question. I think if there are conflicts, we shouldn't use a formatter by default. |
|
To expand on this a little more: We should have a special case for the documented conventions for For the well-known units, we should do a bit more than just appending the unit. We should convert them to an appropriate unit of the same scale, depending on the magnitude of the value. For example, if For units that aren’t well-known, we can just append the unit. I think we should also special-case two other UCUM (non-)units:
We can leave out the special casing of UCUM and do that in a follow-up PR, once elastic/elasticsearch#104037 is merged in which we recommend using UCUM symbols, which we're doing at the moment. |
…4973) ## Summary Default field formatters based on field meta units data. Note: the smallest unit our formatter will show is milliseconds which means micro and nanoseconds may round down to zero for smaller values. #176112 Closes: #82318 Mapping and doc setup for testing - ``` PUT my-index-000001 PUT my-index-000001/_mapping { "properties": { "nanos": { "type": "long", "meta": { "unit": "nanos" } }, "micros": { "type": "long", "meta": { "unit": "micros" } }, "ms": { "type": "long", "meta": { "unit": "ms" } }, "second": { "type": "long", "meta": { "unit": "s" } }, "minute": { "type": "long", "meta": { "unit": "m" } }, "hour": { "type": "long", "meta": { "unit": "h" } }, "day": { "type": "long", "meta": { "unit": "d" } }, "percent": { "type": "long", "meta": { "unit": "percent" } }, "bytes": { "type": "long", "meta": { "unit": "byte" } } } } POST my-index-000001/_doc { "nanos" : 1234.5, "micros" : 1234.5, "ms" : 1234.5, "second" : 1234.5, "minute" : 1234.5, "hour" : 1234.5, "day" : 1234.5, "percent" : 1234.5, "bytes" : 1234.5 } ```
…stic#174973) ## Summary Default field formatters based on field meta units data. Note: the smallest unit our formatter will show is milliseconds which means micro and nanoseconds may round down to zero for smaller values. elastic#176112 Closes: elastic#82318 Mapping and doc setup for testing - ``` PUT my-index-000001 PUT my-index-000001/_mapping { "properties": { "nanos": { "type": "long", "meta": { "unit": "nanos" } }, "micros": { "type": "long", "meta": { "unit": "micros" } }, "ms": { "type": "long", "meta": { "unit": "ms" } }, "second": { "type": "long", "meta": { "unit": "s" } }, "minute": { "type": "long", "meta": { "unit": "m" } }, "hour": { "type": "long", "meta": { "unit": "h" } }, "day": { "type": "long", "meta": { "unit": "d" } }, "percent": { "type": "long", "meta": { "unit": "percent" } }, "bytes": { "type": "long", "meta": { "unit": "byte" } } } } POST my-index-000001/_doc { "nanos" : 1234.5, "micros" : 1234.5, "ms" : 1234.5, "second" : 1234.5, "minute" : 1234.5, "hour" : 1234.5, "day" : 1234.5, "percent" : 1234.5, "bytes" : 1234.5 } ```
…stic#174973) ## Summary Default field formatters based on field meta units data. Note: the smallest unit our formatter will show is milliseconds which means micro and nanoseconds may round down to zero for smaller values. elastic#176112 Closes: elastic#82318 Mapping and doc setup for testing - ``` PUT my-index-000001 PUT my-index-000001/_mapping { "properties": { "nanos": { "type": "long", "meta": { "unit": "nanos" } }, "micros": { "type": "long", "meta": { "unit": "micros" } }, "ms": { "type": "long", "meta": { "unit": "ms" } }, "second": { "type": "long", "meta": { "unit": "s" } }, "minute": { "type": "long", "meta": { "unit": "m" } }, "hour": { "type": "long", "meta": { "unit": "h" } }, "day": { "type": "long", "meta": { "unit": "d" } }, "percent": { "type": "long", "meta": { "unit": "percent" } }, "bytes": { "type": "long", "meta": { "unit": "byte" } } } } POST my-index-000001/_doc { "nanos" : 1234.5, "micros" : 1234.5, "ms" : 1234.5, "second" : 1234.5, "minute" : 1234.5, "hour" : 1234.5, "day" : 1234.5, "percent" : 1234.5, "bytes" : 1234.5 } ```
…stic#174973) ## Summary Default field formatters based on field meta units data. Note: the smallest unit our formatter will show is milliseconds which means micro and nanoseconds may round down to zero for smaller values. elastic#176112 Closes: elastic#82318 Mapping and doc setup for testing - ``` PUT my-index-000001 PUT my-index-000001/_mapping { "properties": { "nanos": { "type": "long", "meta": { "unit": "nanos" } }, "micros": { "type": "long", "meta": { "unit": "micros" } }, "ms": { "type": "long", "meta": { "unit": "ms" } }, "second": { "type": "long", "meta": { "unit": "s" } }, "minute": { "type": "long", "meta": { "unit": "m" } }, "hour": { "type": "long", "meta": { "unit": "h" } }, "day": { "type": "long", "meta": { "unit": "d" } }, "percent": { "type": "long", "meta": { "unit": "percent" } }, "bytes": { "type": "long", "meta": { "unit": "byte" } } } } POST my-index-000001/_doc { "nanos" : 1234.5, "micros" : 1234.5, "ms" : 1234.5, "second" : 1234.5, "minute" : 1234.5, "hour" : 1234.5, "day" : 1234.5, "percent" : 1234.5, "bytes" : 1234.5 } ```
Elasticsearch now has standard metadata entries which we expect to use throughout the stack. The first step to integrating this in Kibana is to store field meta information which is part of field caps in the Kibana Index Pattern object.
cc @mattkime
The text was updated successfully, but these errors were encountered: