Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #17121 to 7.x: [Filebeat] return error when expand_event_list_from_field is missing #17182

Merged
merged 1 commit into from
Mar 23, 2020
Merged

Cherry-pick #17121 to 7.x: [Filebeat] return error when expand_event_list_from_field is missing #17182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ are in JSON format and events are found under the JSON object "Records".
Note: When `expand_event_list_from_field` parameter is given in the config, s3
input will assume the logs are in JSON format and decode them as JSON. Content
type will not be checked.
If a file has "application/json" content-type, `expand_event_list_from_field`
becomes required to read the json file.

[float]
==== `api_timeout`
Expand Down
9 changes: 8 additions & 1 deletion x-pack/filebeat/input/s3/input.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,13 @@ func (p *s3Input) createEventsFromS3Info(svc s3iface.ClientAPI, info s3Info, s3C

reader := bufio.NewReader(resp.Body)

// Check if expand_event_list_from_field is given with document conent-type = "application/json"
if resp.ContentType != nil && *resp.ContentType == "application/json" && p.config.ExpandEventListFromField == "" {
err := errors.New("expand_event_list_from_field parameter is missing in config for application/json content-type file")
p.logger.Error(err)
return err
}

// Decode JSON documents when expand_event_list_from_field is given in config
if p.config.ExpandEventListFromField != "" {
decoder := json.NewDecoder(reader)
Expand All @@ -440,7 +447,7 @@ func (p *s3Input) createEventsFromS3Info(svc s3iface.ClientAPI, info s3Info, s3C
return nil
}

// Check content-type
// Check content-type = "application/x-gzip" or filename ends with ".gz"
if (resp.ContentType != nil && *resp.ContentType == "application/x-gzip") || strings.HasSuffix(info.key, ".gz") {
gzipReader, err := gzip.NewReader(resp.Body)
if err != nil {
Expand Down