Add selective download feature to Data Prepper sources section #6247
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
added
2 - In progress
vagimeli
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
AMoo-Miki,
natebower and
dlvenable
as code owners
vagimeli
added
3 - Tech review
Signed-off-by: Melissa Vagi <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>
|
@dlvenable Please review this use case PR at your availability. I've put it under Data Prepper S3 sources content. If it's not the proper navigation, let me know where you think it best fits. Thank you, Melissa |
Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
commented
Mar 6, 2024
| @@ -0,0 +1,54 @@ | |||
| --- | |||
| layout: default | |||
| title: Selective download | |||
There was a problem hiding this comment.
Should we clarify this title? Something like "Selective download for S3 sources?"
|
@dlvenable At your availability, will you review this use case? It's one of the two last use cases we're transferring from OpenSearch Ingestion. Thank you, Melissa |
|
@dlvenable Following up to get your tech review on this use case carryover from OpenSearch Service Ingestion to Data Prepper. Once we have your approval, I can move this forward to editorial and get it published. |
Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
added
Needs SME
vagimeli
added
5 - Editorial review
natebower
reviewed
Jun 27, 2024
| @@ -20,15 +19,14 @@ The following diagram shows the overall architecture of the components involved. | |||
|
|
|||
| <img src="{{site.url}}{{site.baseurl}}/images/data-prepper/s3-source/s3-architecture.jpg" alt="S3 source architecture">{: .img-fluid} | |||
|
|
|||
| The flow of data is as follows. | |||
| The data flow involving the components is as follows: | |||
There was a problem hiding this comment.
"The data flow involving the components" => "The component data flow"?
| Refer to user guides [Filtering and retrieving data using Amazon S3 Select](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html) and [SQL reference for Amazon S3 Select](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-select-sql-reference.html) for comprehensive information about using Amazon S3 Select. | ||
| {: .note} | ||
|
|
||
| The following example pipeline retrieves all data S3 objects encoded in the Parquet File Format: |
There was a problem hiding this comment.
Either "from" should follow "data", or "data" should be removed.
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
commented
Jun 27, 2024
Signed-off-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
vagimeli
commented
Jun 27, 2024
Signed-off-by: Melissa Vagi <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jun 27, 2024
* Add feature to this section Signed-off-by: Melissa Vagi <[email protected]> * add content Signed-off-by: Melissa Vagi <[email protected]> * Copy edits Signed-off-by: Melissa Vagi <[email protected]> * Update selective-download.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit a673b24) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
leanneeliatra
pushed a commit
to leanneeliatra/opensearch-documentation-website-forl
that referenced
this pull request
Jul 24, 2024
…earch-project#6247) * Add feature to this section Signed-off-by: Melissa Vagi <[email protected]> * add content Signed-off-by: Melissa Vagi <[email protected]> * Copy edits Signed-off-by: Melissa Vagi <[email protected]> * Update selective-download.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: [email protected] <[email protected]>
sandervandegeijn
pushed a commit
to sandervandegeijn/documentation-website
that referenced
this pull request
Jul 30, 2024
…earch-project#6247) * Add feature to this section Signed-off-by: Melissa Vagi <[email protected]> * add content Signed-off-by: Melissa Vagi <[email protected]> * Copy edits Signed-off-by: Melissa Vagi <[email protected]> * Update selective-download.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Address tech review comments Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: David Venable <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add selective download feature to Data Prepper sources section
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.