Support per-field metadata

[jpountz]

It would sometimes be useful for ingestion tools to be able to associate metadata with fields that could later be leveraged by visualization tools such as Kibana to provide a better out-of-the-box experience. One example that got mentioned a number of times for instance is the ability to know the unit of a field. One way to do it would be by giving Elasticsearch the ability to associate metadata per field in the mappings, something like this:

{
  "mappings": {
    "_doc": {
      "properties": {
        "response_time": {
          "type": "float",
          "_meta": {
            "unit": "s"
          }
        },
        "response_size": {
          "type": "long",
          "_meta": {
            "unit": "b"
          }
        }
      }
    }
  }
}

This metadata wouldn't be validated by Elasticsearch: any key-value pairs would be accepted, so there would need to be conventions on key names and values.

Even though things like units are not expected to change on an existing index, preventing updates seems a bit too restrictive since it would be a pity to require reindexing for something that doesn't affect the way that data is indexed. So I propose that updates are merged with existing _meta and that null values may be used to remove existing keys.

Known limitation: For users that already have lots of fields and/or indices, using this feature extensively won't be recommended as it will further increase the size of the cluster state.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jpountz]

Something that just occurred to me is that given that mappings are not available from remote clusters, Kibana would probably want this information to be included in the field capabilities API.

[ruflin]

Mentioning elastic/kibana#44955 and elastic/kibana#35481 from the Kibana side here is I think having this concept could simplify things on the Kibana side.

[mattkime]

I think this is an interesting idea and I'm thinking through the implications.

There's an inherent tension in what we're trying to accomplish. We need to know the field list and associated metadata for fields across potentially thousands of indices. That list is potentially dynamic but static most of the time.

The current solution is for kibana to look up and store the field list. Its reset manually by the user when the field list changes. It would be preferable to close the gap between kibana's view of the field list and the field list in elasticsearch. The lifecycle of the kibana index pattern field mapping better matches that of an index template (random thought).

Would querying field level metadata be scalable?

[robcowart]

This could be very useful if it allowed for new data to enjoy a better initial presentation in Kibana. I would want to see two things...

1. There should be some kind of namespacing, or at least a reserved prefix like kibana_, to avoid conflicts with anything Kibana will use.

2. It must still possible within Kibana Index Patterns to modify/override any attributes autogenerated from this metadata.