Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] - Add raw json field when unmarshaling fails #6591

Merged
merged 7 commits into from
Apr 16, 2018
Merged

[Filebeat] - Add raw json field when unmarshaling fails #6591

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
dc0ab73
add raw json field when json unmarshaling fails in filebeat
crazyvikas23 Mar 19, 2018
0bd3b95
fixed the tests
crazyvikas23 Mar 19, 2018
b71e611
add message key
crazyvikas23 Mar 20, 2018
eedc6f3
add raw json data in message key in main beat event
crazyvikas23 Mar 22, 2018
2372894
remove raw message from error logs so that sensitive data is not comp…
crazyvikas23 Apr 13, 2018
2b1cee5
Update CHANGELOG.asciidoc
andrewkroh Apr 13, 2018
464e1c3
Merge branch 'master' into crazyvikas23/add_raw_json_data_filebeat
andrewkroh Apr 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions filebeat/harvester/reader/json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,11 @@ func (r *JSON) decodeJSON(text []byte) ([]byte, common.MapStr) {
err := unmarshal(text, &jsonFields)
if err != nil || jsonFields == nil {
if !r.cfg.IgnoreDecodingError {
logp.Err("Error decoding JSON: %v", err)

logp.Err("Error decoding JSON: %v. Raw data: %s", err, string(text))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we shouldn't put this raw message into the error logs. I'm worried about sensitive information being leaked inside of the Filebeat logs.

}
if r.cfg.AddErrorKey {
jsonFields = common.MapStr{"error": createJSONError(fmt.Sprintf("Error decoding JSON: %v", err))}
jsonFields = common.MapStr{"error": createJSONError(fmt.Sprintf("Error decoding JSON: %v. Raw data: %s", err, string(text)))}
}
return text, jsonFields
}
Expand Down
4 changes: 2 additions & 2 deletions filebeat/harvester/reader/json_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,14 +125,14 @@ func TestDecodeJSON(t *testing.T) {
Text: `null`,
Config: JSONConfig{MessageKey: "value", AddErrorKey: true},
ExpectedText: `null`,
ExpectedMap: common.MapStr{"error": common.MapStr{"message": "Error decoding JSON: <nil>", "type": "json"}},
ExpectedMap: common.MapStr{"error": common.MapStr{"message": "Error decoding JSON: <nil>. Raw data: null", "type": "json"}},
},
{
// Add key error helps debugging this
Text: `{"message": "test", "value": "`,
Config: JSONConfig{MessageKey: "value", AddErrorKey: true},
ExpectedText: `{"message": "test", "value": "`,
ExpectedMap: common.MapStr{"error": common.MapStr{"message": "Error decoding JSON: unexpected EOF", "type": "json"}},
ExpectedMap: common.MapStr{"error": common.MapStr{"message": `Error decoding JSON: unexpected EOF. Raw data: {"message": "test", "value": "`, "type": "json"}},
},
{
// If the text key is not found, put an error
Expand Down