Metricbeats Kubernetes event module unable to parse empty date field

[mikelorant]

Issue
Getting the following error when trying to parse an event.

2017-09-04T02:11:28Z WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [kubernetes.event.metadata.timestamp.deleted]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"\""}}

Enabling debug mode showed the following event message.

2017-09-04T02:27:52Z DBG  Publish event: {
  "@timestamp": "2017-09-04T02:27:52.271Z",
  "@metadata": {
    "beat": "\u003cnot set\u003e",
    "type": "doc"
  },
  "kubernetes": {
    "event": {
      "message": "Created pod: ywqvd-kube-state-metrics-483864468-x39ft",
      "reason": "SuccessfulCreate",
      "type": "Normal",
      "count": 1,
      "involved_object": {
        "kind": "ReplicaSet",
        "uid": "86ada97c-9107-11e7-81a9-024fe15315e7",
        "api_version": "extensions",
        "resource_version": "19155721",
        "name": "ywqvd-kube-state-metrics-483864468"
      },
      "metadata": {
        "generate_name": "",
        "uid": "a6da2702-9118-11e7-81a9-024fe15315e7",
        "resource_version": "367834",
        "timestamp": {
          "created": "2017-09-04T02:27:52Z",
          "deleted": ""
        },
        "name": "ywqvd-kube-state-metrics-483864468.14e107918ea3c588",
        "namespace": "infrastructure-metrics-shared-v1",
        "self_link": "/api/v1/namespaces/infrastructure-metrics-shared-v1/events/ywqvd-kube-state-metrics-483864468.14e107918ea3c588"
      },
      "timestamp": {
        "first_occurrence": "2017-09-04T02:27:52.000Z",
        "last_occurrence": "2017-09-04T02:27:52.000Z"
      }
    }
  },
  "metricset": {
    "module": "kubernetes",
    "name": "event"
  },
  "beat": {
    "name": "ywqvd-metricbeat-4231345040-qks11",
    "hostname": "ywqvd-metricbeat-4231345040-qks11",
    "version": "6.0.0-beta2"
  },
  "meta": {
    "cloud": {
      "instance_id": "i-0f919437f290ba006",
      "machine_type": "r4.2xlarge",
      "region": "ap-southeast-2",
      "availability_zone": "ap-southeast-2a",
      "provider": "ec2"
    }
  }
}

The deleted timestamp field is empty.

Looking at the source code it is clear that the field is marked as date and does not expect to be empty.
https://github.com/elastic/beats/blob/master/metricbeat/module/kubernetes/event/_meta/fields.yml#L48

@vjsamuel identified that the root cause is likely the field has been set to string.
https://github.com/elastic/beats/blob/master/metricbeat/module/kubernetes/event/types.go#L8

[exekias]

Thank you for the report! What version of Metricbeat are you using?

[mikelorant]

We are using metricbeat 6.0.0 beta 2.