[Filebeat][Gsuite] Transform all dates to timestamp with processor #20308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/siem (Team:SIEM) |
botelastic
bot
added
needs_team
adriansr
approved these changes
Jul 29, 2020
adriansr
reviewed
Jul 29, 2020
Comment on lines
450
to
471
| var parseDate = function(field, targetField) { | ||
| return function(evt) { | ||
| new processor.Timestamp({ | ||
| field: field, | ||
| target_field: targetField, | ||
| timezone: "UTC", | ||
| layouts: [ | ||
| "2006-01-02T15:04:05Z", | ||
| "2006-01-02T15:04:05.999Z", | ||
| "2006/01/02 15:04:05 UTC", | ||
| ], | ||
| tests: [ | ||
| "2020-02-05T18:19:23Z", | ||
| "2020-02-05T18:19:23.599Z", | ||
| "2020/07/28 04:59:59 UTC", | ||
| ], | ||
| ignore_missing: true, | ||
| }).Run(evt); | ||
|
|
||
| evt.Delete(field); | ||
| }; | ||
| }; |
There was a problem hiding this comment.
Nit:
This pattern is allocating a new Timestamp processor for each event it processes. I think it would make more sense to create a Chain in the outer function.
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Jul 29, 2020
…lastic#20308) * Transform all dates to timestamp with processor * Change parse date function to create a chain (cherry picked from commit 953d067)
marc-gr
added a commit
to marc-gr/beats
that referenced
this pull request
Jul 29, 2020
…lastic#20308) * Transform all dates to timestamp with processor * Change parse date function to create a chain (cherry picked from commit 953d067)
marc-gr
added a commit
that referenced
this pull request
Jul 29, 2020
marc-gr
added a commit
that referenced
this pull request
Jul 29, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 30, 2020
…ne-2.0 * upstream/master: (29 commits) Add an explicit system test for processes on unix systems (elastic#20320) [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks (elastic#20305) [CI] Update README.md with CI references (elastic#20316) Add ECK doc links to Heartbeat docs (elastic#20284) [Filebeat] Add export tests to x-pack/filebeat (elastic#20156) feat(ci): support building docker images for PRs (elastic#20323) Update system tests dependency (elastic#20287) [Libbeat] Log debug message if the Kibana dashboard can not be imported from the archive (elastic#12211) (elastic#20150) [Filebeat][Gsuite] Transform all dates to timestamp with processor (elastic#20308) Infer types in Prometheus remote_write (elastic#19944) Remove unnecessary restarts of metricsets while using Node autodiscover (elastic#19974) docs: update changelog on master branch (elastic#20259) feat(ci): support storing artifacts for PRs in separate dirs (elastic#20282) [CI] Change upstream reference (elastic#20296) [Filebeat] Updates to Suricata module (elastic#20220) [docs] Fix Windows download link for agent (elastic#20258) [docs] Rename release highlights to what's new (elastic#20255) fix: update the display name of the multibranch job (elastic#20265) [Elastic Agent] Add basic protocol to control Elastic Agent. (elastic#20146) Cisco ASA: Fix message 106100 (elastic#20245) ...
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
…lastic#20308) * Transform all dates to timestamp with processor * Change parse date function to create a chain
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…lastic#20308) (elastic#20319) * Transform all dates to timestamp with processor * Change parse date function to create a chain (cherry picked from commit 8eb5d14)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Parse properly date time fields in different formats in gsuite module.
Why is it important?
Not all fields come in the same format, and some of them were not accepted as valid dates. This change proactively parses all date fields with the known incoming formats.
Checklist
- [ ] I have commented my code, particularly in hard-to-understand areas- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Logs
Logs that did show the bug