Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Event Consistency via metadata and JSON changes #1803

Merged
merged 2 commits into from
Sep 26, 2022

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented Sep 24, 2022

Description

This makes a few end-user changes related to the S3 source:

  • Moves the key and bucket keys into a metadata key that starts with s3/ by default.
  • Adds the metadata_root_key S3 source config which allows modifying the key specified above.
  • Changes the JSON codec to populate the root Event rather than the message key.

Along with this change, I did some refactoring to decouple the metadata population from the S3 object processing.

Issues Resolved

Resolves #1687

Check List

  • New functionality includes testing.
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dlvenable dlvenable requested a review from a team as a code owner September 24, 2022 21:22
@codecov-commenter
Copy link

codecov-commenter commented Sep 24, 2022

Codecov Report

Merging #1803 (bade15d) into main (504e8b8) will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #1803      +/-   ##
============================================
- Coverage     93.96%   93.84%   -0.12%     
  Complexity     1457     1457              
============================================
  Files           188      188              
  Lines          4275     4275              
  Branches        347      347              
============================================
- Hits           4017     4012       -5     
- Misses          177      182       +5     
  Partials         81       81              
Impacted Files Coverage Δ
.../org/opensearch/dataprepper/pipeline/Pipeline.java 88.40% <0.00%> (-4.35%) ⬇️
...rwarder/discovery/AwsCloudMapPeerListProvider.java 92.59% <0.00%> (-2.47%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Collaborator

@oeyh oeyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@dlvenable dlvenable merged commit 6d35560 into opensearch-project:main Sep 26, 2022
@dlvenable dlvenable deleted the 1687-s3-metadata-key branch October 5, 2022 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

S3 Event Decoding Consistency
4 participants