Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ibmmq: Fix timestamp parsing #29773

Merged
merged 1 commit into from
Jan 11, 2022
Merged

ibmmq: Fix timestamp parsing #29773

merged 1 commit into from
Jan 11, 2022

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Jan 10, 2022 •
edited
Loading

What does this PR do?

Fixes the @timestamp parsing in Filebeat's ibmmq module. It had a the following problems:

  • Date processor format definition was broken (extra a character), which caused the processor to always fail.
  • The header date format in some logs was unsupported (dd.MM.yyyy).
  • The Time() field, with correct TZ and higher precision, was ignored.

Now it will correctly parse the header timestamp for all logs, and use the better Time() field if available.

Why is it important?

For proper parsing of dates.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@adriansr adriansr added in progress Pull request is currently in progress. Team:Integrations Label for the Integrations team backport-v8.0.0 Automated backport with mergify labels Jan 10, 2022
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 10, 2022
@adriansr adriansr mentioned this pull request Jan 10, 2022
Merged
3 tasks
@adriansr adriansr added the Filebeat Filebeat label Jan 10, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 10, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-01-10T16:06:27.683+0000

  • Duration: 103 min 40 sec

  • Commit: 68e0623

Test stats 🧪

Test Results
Failed 0
Passed 9644
Skipped 1282
Total 10926

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@adriansr
ibmmq: Fix timestamp parsing
68e0623 
This fixes the timestamp parsing in ibmmq logs:
- Date processor format definition was broken for ES 8.0 (extra `a`
  character).
- The header date format in some logs was unsupported.
- The Time() field, with correct TZ and higher precission was ignored.
@adriansr adriansr force-pushed the ibmmq_errorlog_timestamp_fix branch from 6563c2e to 68e0623 Compare January 10, 2022 16:06
@adriansr adriansr marked this pull request as ready for review January 10, 2022 16:16
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@adriansr adriansr added review and removed in progress Pull request is currently in progress. labels Jan 10, 2022
@adriansr adriansr merged commit fbc33ab into elastic:master Jan 11, 2022
@adriansr adriansr deleted the ibmmq_errorlog_timestamp_fix branch January 11, 2022 09:30
@mergify mergify bot mentioned this pull request Jan 11, 2022
Merged
mergify bot pushed a commit that referenced this pull request Jan 11, 2022
@adriansr
ibmmq: Fix timestamp parsing (#29773)
ee61cac 
This fixes the timestamp parsing in ibmmq logs:
- Date processor format definition was broken for ES 8.0 (extra `a`
  character).
- The header date format in some logs was unsupported.
- The Time() field, with correct TZ and higher precission was ignored.

(cherry picked from commit fbc33ab)
adriansr added a commit that referenced this pull request Jan 11, 2022
@mergify @adriansr
[8.0](backport #29773) ibmmq: Fix timestamp parsing (#29788)
b32aa6a 
* ibmmq: Fix timestamp parsing (#29773)

This fixes the timestamp parsing in ibmmq logs:
- Date processor format definition was broken for ES 8.0 (extra `a`
  character).
- The header date format in some logs was unsupported.
- The Time() field, with correct TZ and higher precission was ignored.

(cherry picked from commit fbc33ab)

Co-authored-by: Adrian Serrano <[email protected]>
adriansr added a commit to adriansr/beats that referenced this pull request Jan 11, 2022
@adriansr
Missing changelog entry for elastic#29773
040b925 
Relates elastic#29773
@adriansr adriansr mentioned this pull request Jan 11, 2022
Merged
adriansr added a commit that referenced this pull request Jan 11, 2022
@adriansr
Missing changelog entry for #29773 (#29791)
7a0eb7c 
Relates #29773
v1v added a commit to v1v/beats that referenced this pull request Jan 12, 2022
@v1v
Merge remote-tracking branch 'upstream/master' into feature/add-githu…
0bab1d5 
…b-for-macos

* upstream/master: (172 commits)
  [Elastic Agent] Fix issue with ensureServiceToken. (elastic#29800)
  [Winlogbeat] Add provider name to Security routing pipeline check (elastic#29781)
  Add summary to journeys which don't emit journey:end (early node subprocess exits) (elastic#29606)
  Prepare 8.0.0-rc1 changelog (elastic#29795) (elastic#29806)
  Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29681)
  libbeat/processors/add_process_metadata: implement a process cache eviction policy (elastic#29717)
  [Automation] Update elastic stack version to 8.1.0-7004acda for testing (elastic#29783)
  Missing changelog entry for elastic#29773 (elastic#29791)
  Add a readme for k8s autodiscover provider (elastic#28213)
  Remove overriding of index pattern on the Kubernetes overview dashboard (elastic#29676)
  jjbb: remove obsoleted branches (<7.16) (elastic#29707)
  Add k8s metadata in state_cronjob metricset (elastic#29572)
  ibmmq: Fix timestamp parsing (elastic#29773)
  Do not add date to index if `@meta.index` is set (elastic#29775)
  ci: uses aliases for the branches (elastic#29706)
  Filebeat tests: Restore `@timestamp` field validation (elastic#29772)
  Forward port 7.16.3 changelog to master (elastic#29777)
  auditd: Store program arguments in process.args array (elastic#29601)
  System/socket: Support kernel_clone() replacement for _do_fork() (elastic#29744)
  Do not mention removal if version is not specified in `cfgwarn` messages (elastic#29727)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees
No one assigned
Labels
backport-v8.0.0 Automated backport with mergify Filebeat Filebeat review Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adriansr @elasticmachine @kaiyan-sheng