metric_type does not work for all the expected fields

[ruflin]

I'm currently running 8.7.0-SNAPSHOT and played around with the TSDB metrics fields. I modified the integration package and added the metric_type: gauge and counter to various fields but the outcome is not persistent. Looking closer, it looks like time_series_metric is only applied to fields that are scaled_float but it should apply to a much broader range of fields: https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html#time-series-metric Looking at the code, it seems the time_series_metric part is only implemented for scaled_float. This was implemented in #126322

Few questions:

• What is the reason this was only added for scaled_floats?
• Looking at the mappings, meta information + mappings is added. Why is that?

Examples

The fields .yml looks as following:

- name: swap
 type: group
 fields:
   - name: total
     type: long
     format: bytes
     unit: byte
     metric_type: gauge
     description: |
       Total swap memory.
   - name: used.bytes
     type: long
     format: bytes
     unit: byte
     metric_type: gauge
     description: |
       Used swap memory.
   - name: free
     type: long
     format: bytes
     unit: byte
     metric_type: gauge
     description: |
       Available swap memory.
   - name: out.pages
     type: long
     metric_type: counter
     description: count of pages swapped out
   - name: in.pages
     type: long
     metric_type: gauge
     description: count of pages swapped in
   - name: readahead.pages
     type: long
     metric_type: counter
     description: swap readahead pages
   - name: readahead.cached
     type: long
     description: swap readahead cache hits
   - name: used.pct
     type: scaled_float
     format: percent
     unit: percent
     metric_type: gauge
     description: |
       The percentage of used swap memory.

The resulting json template:

"swap": {
  "properties": {
    "total": {
      "meta": {
        "unit": "byte",
        "metric_type": "gauge"
      },
      "type": "long"
    },
    "readahead": {
      "properties": {
        "pages": {
          "meta": {
            "metric_type": "counter"
          },
          "type": "long"
        },
        "cached": {
          "type": "long"
        }
      }
    },
    "in": {
      "properties": {
        "pages": {
          "meta": {
            "metric_type": "gauge"
          },
          "type": "long"
        }
      }
    },
    "used": {
      "properties": {
        "pct": {
          "time_series_metric": "gauge",
          "meta": {
            "unit": "percent",
            "metric_type": "gauge"
          },
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "bytes": {
          "meta": {
            "unit": "byte",
            "metric_type": "gauge"
          },
          "type": "long"
        }
      }
    },
    "free": {
      "meta": {
        "unit": "byte",
        "metric_type": "gauge"
      },
      "type": "long"
    },
    "out": {
      "properties": {
        "pages": {
          "meta": {
            "metric_type": "counter"
          },
          "type": "long"
        }
      }
    }
  }
},

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

cc @juliaElastic @nchaulet - I don't have a great understanding of the specifics for the scaled_float only implementation here. Could either of you take a look at @ruflin's questions above?

[juliaElastic]

What is the reason this was only added for scaled_floats?

I wasn't aware that more types support time_series_metric, we can fix this.

Looking at the mappings, meta information + mappings is added. Why is that?

@ruflin I'm not really sure what do you mean, could you clarify?

[ruflin]

I'm not really sure what do you mean, could you clarify?

Under meta, the gauge information is repeated with metric_type: gauge. Is this used anywhere? It might be useful if time_series is not enabled to indicate it is a gauge but in case time_series is enabled, it looks like duplicated info.

"pct": {
  "time_series_metric": "gauge",
  "meta": {
    "unit": "percent",
    "metric_type": "gauge"
  },
  "scaling_factor": 1000,
  "type": "scaled_float"
},

[juliaElastic]

Under meta, the gauge information is repeated with metric_type: gauge. Is this used anywhere? It might be useful if time_series is not enabled to indicate it is a gauge but in case time_series is enabled, it looks like duplicated info.

Ah yes, meta.metric_type can be removed, I think. The original feature request mentioned to keep meta.unit only: #115621