Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Updated GZipCompressionEngine to use GzipCompressorInputStream #1570

Merged
merged 1 commit into from
Jul 8, 2022

Conversation

asifsmohammed
Copy link
Collaborator

@asifsmohammed asifsmohammed commented Jul 7, 2022

Signed-off-by: Asif Sohail Mohammed [email protected]

Description

This fixes the bug where all the log files are not coming through when logs are new line delimited with GZip compression.
Using apache-commons compress library to fix this by replacing GZIPInputStream with GzipCompressorInputStream.

➡️ There will be a separate PR for Integration Testing.

Issues Resolved

Resolves #1568

Check List

  • New functionality includes testing.
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@asifsmohammed asifsmohammed requested a review from a team as a code owner July 7, 2022 20:01
Copy link
Collaborator

@chenqi0805 chenqi0805 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

@dlvenable dlvenable merged commit bde1b52 into opensearch-project:main Jul 8, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 8, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 8, 2022
asifsmohammed added a commit that referenced this pull request Jul 8, 2022
…1570) (#1571)

Signed-off-by: Asif Sohail Mohammed <[email protected]>
(cherry picked from commit bde1b52)

Co-authored-by: Asif Sohail Mohammed <[email protected]>
asifsmohammed added a commit that referenced this pull request Jul 8, 2022
…1570) (#1572)

Signed-off-by: Asif Sohail Mohammed <[email protected]>
(cherry picked from commit bde1b52)

Co-authored-by: Asif Sohail Mohammed <[email protected]>
engechas pushed a commit to engechas/data-prepper that referenced this pull request Sep 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] S3 Source fails to load all records for large compressed newline-delimited logs
3 participants