Fix nil panic when overwriting metadata #24741

sincejune · 2021-03-24T15:43:21Z

What does this PR do?

I'm using Filebeat writing data into Kafka and having another Filebeat reading from Kafka. I got this error when I was trying to decode the message field into root-level via decode_json_fields processor.

panic: assignment to entry in nil map

goroutine 105 [running]:
github.com/elastic/beats/libbeat/common.MapStr.Update(0x0, 0xc0001620c0)
	/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:61 +0xd3
github.com/elastic/beats/libbeat/common/jsontransform.WriteJSONKeys(0xc0000b8080, 0xc000162090, 0x101)
	/go/src/github.com/elastic/beats/libbeat/common/jsontransform/jsonhelper.go:67 +0x9ad
github.com/elastic/beats/libbeat/processors/actions.(*decodeJSONFields).Run(0xc0001710c0, 0xc0000b8080, 0xc00017bb00, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/processors/actions/decode_json_fields.go:145 +0x751
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000768d50, 0xc0000b8080, 0xc0000b8080, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000768d20, 0xc0000b8080, 0x203000, 0x7f10562985b0, 0x7f1056295ff8)
	/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc000184f80, 0xe3ed9c0, 0xed7ed39c7, 0x3da5020, 0x0, 0xc00017abd0, 0x2551760, 0xc0001b8410, 0x0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x571
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc000184f80, 0xe3ed9c0, 0xed7ed39c7, 0x3da5020, 0x0, 0xc00017abd0, 0x2551760, 0xc0001b8410, 0x0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2
github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc000768f60, 0xe3ed9c0, 0xed7ed39c7, 0x3da5020, 0x0, 0xc00017abd0, 0x2551760, 0xc0001b8410, 0x0, 0x0)
	/go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e
github.com/elastic/beats/filebeat/input/kafka.(*groupHandler).ConsumeClaim(0xc000164000, 0x2a49a00, 0xc0007b6080, 0x2a3cd60, 0xc000495c80, 0xa, 0x0)
	/go/src/github.com/elastic/beats/filebeat/input/kafka/input.go:332 +0x1c5
github.com/elastic/beats/vendor/github.com/Shopify/sarama.(*consumerGroupSession).consume(0xc0007b6080, 0xc00027f380, 0xa, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/Shopify/sarama/consumer_group.go:615 +0x27f
github.com/elastic/beats/vendor/github.com/Shopify/sarama.newConsumerGroupSession.func2(0xc0007b6080, 0xc00027f380, 0xa, 0xc000000000)
	/go/src/github.com/elastic/beats/vendor/github.com/Shopify/sarama/consumer_group.go:544 +0xd0
created by github.com/elastic/beats/vendor/github.com/Shopify/sarama.newConsumerGroupSession
	/go/src/github.com/elastic/beats/vendor/github.com/Shopify/sarama/consumer_group.go:536 +0x526

Why is it important?

The Filebeat process will crash without the fix.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

using unittest.

Related issues

Closes json.overwrite_keys causes filebeat to panic #6381

Note

Filebeat will actually output JSONs with duplicated keys when I'm trying to overwrite the @metadata field. And I'm not pretty sure if that's good for beats. Check the example below:

Filebeat configuration

filebeat.inputs:
  - type: stdin
    enabled: true
output.console:
    enabled: true

processors:
  - decode_json_fields:
      fields: [ "message" ]
      target: ""
      overwrite_keys: true

Input

{"@timestamp":"2021-03-24T12:17:54.978Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.0.0"},"message":"overwrite metadata test"}

Output

{"@timestamp":"2021-03-24T12:17:54.978Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.0.0","type":"_doc","version":"7.0.0","beat":"filebeat"},"log":{"offset":0,"file":{"path":""}},"message":"overwrite metadata test","input":{"type":"stdin"},"agent":{"name":"MacBook","type":"filebeat","version":"8.0.0","ephemeral_id":"ephemeral_id","id":"id"},"ecs":{"version":"1.8.0"},"host":{"name":"MacBook"}}

elasticmachine · 2021-03-24T16:53:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: jsoriano commented: /test
Start Time: 2021-04-20T17:18:44.142+0000
Duration: 61 min 38 sec
Commit: bb93773

Test stats 🧪

Test	Results
Failed	0
Passed	1862
Skipped	149
Total	2011

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1862
Skipped	149
Total	2011

elasticmachine · 2021-03-29T19:06:16Z

Pinging @elastic/integrations (Team:Integrations)

jsoriano · 2021-03-29T19:06:43Z

/test

elasticmachine · 2021-03-29T19:07:07Z

Pinging @elastic/agent (Team:Agent)

mergify · 2021-04-07T16:27:48Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-panic-overwriting-metadata upstream/fix-panic-overwriting-metadata
git merge upstream/master
git push upstream fix-panic-overwriting-metadata

jsoriano · 2021-04-12T14:16:31Z

/test

libbeat/common/jsontransform/jsonhelper.go

mergify · 2021-04-12T15:30:33Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-panic-overwriting-metadata upstream/fix-panic-overwriting-metadata
git merge upstream/master
git push upstream fix-panic-overwriting-metadata

Co-authored-by: Jaime Soriano Pastor <[email protected]>

jsoriano · 2021-04-15T09:54:31Z

/test

jsoriano

Waiting for green, thanks!

jsoriano · 2021-04-15T13:09:12Z

/test

mergify · 2021-04-16T20:25:43Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-panic-overwriting-metadata upstream/fix-panic-overwriting-metadata
git merge upstream/master
git push upstream fix-panic-overwriting-metadata

jsoriano · 2021-04-20T17:18:24Z

/test

Fix panic when overwriting metadata using the decode_json_fields processor. (cherry picked from commit ce680e1)

Fix panic when overwriting metadata using the decode_json_fields processor. (cherry picked from commit ce680e1) Co-authored-by: Chao <[email protected]>

jsoriano · 2021-04-21T14:38:36Z

@sincejune thanks a lot for this fix! I merged it and it will be available in 7.13.

sincejune · 2021-04-21T14:55:31Z

@jsoriano You're welcome. Thanks for taking care of this!

…-github-pr-comment-template * upstream/master: Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113)

…ng-versions-stack * upstream/master: (28 commits) Add support for parsers in filestream input (elastic#24763) Skip flaky test TestFilestreamTruncate (elastic#25218) backport: Add 7.13 branch (elastic#25189) Update decode_json_fields.asciidoc (elastic#25056) [Elastic Agent] Fix status and inspect command to work inside running container (elastic#25204) Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113) [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) ...

fix nil panic when overwriting metadata

3063a0d

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2021

update CHANGELOG

9f56025

jsoriano added the Team:Integrations Label for the Integrations team label Mar 29, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2021

jsoriano added the Team:Elastic-Agent Label for the Agent team label Mar 29, 2021

jsoriano added the review label Mar 29, 2021

Merge branch 'master' into fix-panic-overwriting-metadata

c1626a1

jsoriano reviewed Apr 12, 2021

View reviewed changes

libbeat/common/jsontransform/jsonhelper.go Outdated Show resolved Hide resolved

sincejune and others added 3 commits April 15, 2021 17:21

Update libbeat/common/jsontransform/jsonhelper.go

0f2d061

Co-authored-by: Jaime Soriano Pastor <[email protected]>

update event.Meta in another case

c95f2c6

Merge branch 'master' into fix-panic-overwriting-metadata

65308c4

jsoriano approved these changes Apr 15, 2021

View reviewed changes

sincejune and others added 2 commits April 17, 2021 13:26

Merge branch 'master' into fix-panic-overwriting-metadata

3694925

Merge remote-tracking branch 'origin/master' into HEAD

bb93773

jsoriano merged commit ce680e1 into elastic:master Apr 20, 2021

jsoriano mentioned this pull request Apr 20, 2021

Cherry-pick #24741 to 7.x: Fix nil panic when overwriting metadata #25180

Merged

6 tasks

jsoriano added the v7.13.0 label Apr 20, 2021

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Apr 20, 2021

Fix nil panic when overwriting metadata (elastic#24741)

fa88da8

Fix panic when overwriting metadata using the decode_json_fields processor. (cherry picked from commit ce680e1)

jsoriano added a commit that referenced this pull request Apr 20, 2021

Fix nil panic when overwriting metadata (#24741) (#25180)

82e1226

Fix panic when overwriting metadata using the decode_json_fields processor. (cherry picked from commit ce680e1) Co-authored-by: Chao <[email protected]>

sincejune deleted the fix-panic-overwriting-metadata branch April 21, 2021 04:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix nil panic when overwriting metadata #24741

Fix nil panic when overwriting metadata #24741

sincejune commented Mar 24, 2021 •

edited

Loading

elasticmachine commented Mar 24, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Mar 29, 2021

jsoriano commented Mar 29, 2021

elasticmachine commented Mar 29, 2021

mergify bot commented Apr 7, 2021

jsoriano commented Apr 12, 2021

mergify bot commented Apr 12, 2021

jsoriano commented Apr 15, 2021

jsoriano left a comment

jsoriano commented Apr 15, 2021

mergify bot commented Apr 16, 2021

jsoriano commented Apr 20, 2021

jsoriano commented Apr 21, 2021

sincejune commented Apr 21, 2021

Fix nil panic when overwriting metadata #24741

Fix nil panic when overwriting metadata #24741

Conversation

sincejune commented Mar 24, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Note

Filebeat configuration

Input

Output

elasticmachine commented Mar 24, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Mar 29, 2021

jsoriano commented Mar 29, 2021

elasticmachine commented Mar 29, 2021

mergify bot commented Apr 7, 2021

jsoriano commented Apr 12, 2021

mergify bot commented Apr 12, 2021

jsoriano commented Apr 15, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Apr 15, 2021

mergify bot commented Apr 16, 2021

jsoriano commented Apr 20, 2021

jsoriano commented Apr 21, 2021

sincejune commented Apr 21, 2021

sincejune commented Mar 24, 2021 •

edited

Loading

elasticmachine commented Mar 24, 2021 •

edited by jenkins-beats-ci bot

Loading