Error ingesting Office 365 events due to invalid IP address field

[adriansr]

For confirmed bugs, please report:

• Version: 7.7.0
• Operating System: n/a
• Discuss Forum URL: https://discuss.elastic.co/t/office-365-module-not-an-ip-string-literal
• Steps to Reproduce: n/a

Some (Exchange audit?) events from Office 365 Management API are causing an ingestion error due to an invalid IP address field (client.ip, maybe others).

May 14 16:51:23 ainfcp1esl00001 filebeat: 2020-05-14T16:51:23.611-0700#011WARN#011[elasticsearch]#011elasticsearch/client.go:384#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63724844335, loc:(*time.Location)(nil)}, Meta:{"_id":"17efaaf6-8bb5-4829-27b1-08d7f6153bf3","pipeline":"filebeat-7.7.0-o365-audit-pipeline"}, Fields:{"agent":{"ephemeral_id":"c7b7e853-c8ba-4749-80e0-b52d98db95ed","hostname":"hostname.com","id":"9ff5gad7-a611-4443-bc73-3d377f156e21","type":"filebeat","version":"7.7.0"},"client":{"address":"[1.2.2.3]:7972","ip":"[1.2.2.3]","port":"7972"},"domain":"domain.com","ecs":{"version":"1.5.0"},"error":{"message":"GoError: cannot override existing key with client.ip"},"event":{"action":"Create","category":"web","code":"ExchangeItem","dataset":"o365.audit","id":"17efaag6-7bb5-4829-27b1-08d7f6153bf3","kind":"event","module":"o365","outcome":"success","provider":"Exchange","type":"info"},"fields":{},"fileset":{"name":"audit"},"host":{"architecture":"x86_64","containerized":false,"hostname":"hostname.com","id":"a962373b0c22485eae53ee6e404cd357","ip":["10.1.2.200"fe80::e75a:ad3:88ff:1ad0"],"mac":["00:50:56:a7:f8:4d"],"name":"hostname.com","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-1127.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"input":{"type":"o365audit"},"o365audit":{"ClientIP":"[1.2.2.3]:7972","ClientIPAddress":"[1.2.2.3]:7972","ClientInfoString":"Client=MSExchangeRPC","ClientProcessName":"OUTLOOK.EXE","ClientVersion":"16.0.12730.20144"
...
Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [client.ip] of type [ip] in document with id '17efaaf6-8bb5-4829-27b1-08d7f6153bf3'. Preview of field's value: '[1.2.2.3]'","caused_by":{"type":"illegal_argument_exception","reason":"'[1.2.2.3]' is not an IP string literal."}}

The important error messages being:

(error.message): GoError: cannot override existing key with client.ip

and

{"type":"mapper_parsing_exception","reason":"failed to parse field [client.ip] of type [ip]"
"caused_by":{"type":"illegal_argument_exception","reason":"'[1.2.2.3]' is not an IP string literal."}}

Caused by the following fields in the original O365 event:

"ClientIP":"[10.1.1.3]:7972",
"ClientIPAddress":"[10.1.1.3]:7972"

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)