[metricbeat] kubernetes.event.message should be type string

[Feder1co5oave]

I collect and index kubernetes events with a singleton metricbeat agent and the kubernetes module (version 6.7.1), with metricset event. I created a kibana discovery search (which I can contribute if requested) through which I can easily read the stream of events.
It looks like this:

As you can see, the kubernetes.event.message field is a human readable phrase that describes the event using words. Problem is, it is mapped to a keyword field (see fieldlist) and I cannot do fulltext searches on it. I think it should be mapped to a string field in fields.yml.

edit: I'm seeing now that this should have been already addressed in PR #10284, but I confirm a cannot see the message field in any document indexed to ES:

{
  "_index": "metrics-2019.09.10",
  "_type": "doc",
  "_id": "43S0Gm0B7z4iRZMrozvy",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-09-10T10:25:08.123Z",
    "event": {
      "dataset": "kubernetes.event"
    },
    "kubernetes": {
      "event": {
        "involved_object": {
          "uid": "42479490-d3b5-11e9-b45c-a4bf0120a378",
          "api_version": "v1",
          "resource_version": "289719589",
          "name": "kylin-cube-refresh-1568111100-b9mpn",
          "kind": "Pod"
        },
        "metadata": {
          "timestamp": {
            "created": "2019-09-10T10:25:08.000Z"
          },
          "name": "kylin-cube-refresh-1568111100-b9mpn.15c30cbe9af4cf47",
          "namespace": "hadoop",
          "self_link": "/api/v1/namespaces/hadoop/events/kylin-cube-refresh-1568111100-b9mpn.15c30cbe9af4cf47",
          "generate_name": "",
          "uid": "432cebc9-d3b5-11e9-9c17-a4bf0120289e",
          "resource_version": "289719615"
        },
        "timestamp": {
          "first_occurrence": "2019-09-10T10:25:08.000Z",
          "last_occurrence": "2019-09-10T10:25:08.000Z"
        },
        "message": "Created container",
        "reason": "Created",
        "type": "Normal",
        "count": 1
      }
    },
    "metricset": {
      "name": "event",
      "module": "kubernetes",
      "host": "https://kubernetes.default:443"
    },
    "beat": {
      "version": "6.7.1",
      "name": "metricbeat-cluster-db7df9b8b-n7s87",
      "hostname": "metricbeat-cluster-db7df9b8b-n7s87"
    },
    "host": {
      "name": "metricbeat-cluster-db7df9b8b-n7s87"
    }
  },
  "fields": {
    "kubernetes.event.timestamp.first_occurrence": [
      "2019-09-10T10:25:08.000Z"
    ],
    "kubernetes.event.timestamp.last_occurrence": [
      "2019-09-10T10:25:08.000Z"
    ],
    "kubernetes.event.metadata.timestamp.created": [
      "2019-09-10T10:25:08.000Z"
    ],
    "@timestamp": [
      "2019-09-10T10:25:08.123Z"
    ]
  },
  "highlight": {
    "metricset.name": [
      "@kibana-highlighted-field@event@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1568111108123
  ]
}

I initially uploaded the metricbeat templates to ES with metricbeat setup --template -e -E setup.template.enabled=true.
I'm wondering what I'm doing wrong or if the PR is not doing what I think it does.

[ChrsMark]

Hi @Feder1co5oave! Sorry for the confusion here!

First of all you are right that we have to change the field type to text, PR already opened.

As far as the copy_to is concerned, it does what is expected but it is a little bit confusing. copy_to only modifies the indexed document, not the source document, so this is why you cannot see that field in the original event. Also, this copy_to is the reason you can already perform "full text search" from Kibana (on message field) but you cannot see the results to be highlighted since the field is not listed. You can find more on https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html.

Let me know what you think!

[Feder1co5oave]

Hi @ChrsMark, thanks for the clear explanation and the quick fix!
I was in fact a bit confused by the message field, which was not visible in kibana. I'm glad this was solved, and I even took the opportunity to learn something new about elasticsearch, that is thecopy_to parameter :D

Cheers!