Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update to use the opensearch-java client for _bulk requests #1347

Closed
Tracked by #1311
dlvenable opened this issue Apr 29, 2022 · 2 comments · Fixed by #1381
Closed
Tracked by #1311

Update to use the opensearch-java client for _bulk requests #1347

dlvenable opened this issue Apr 29, 2022 · 2 comments · Fixed by #1381
Assignees
@dlvenable
Milestone
v1.4

Comments

@dlvenable
Copy link
Member

dlvenable commented Apr 29, 2022 •
edited
Loading

Update Data Prepper to use the opensearch-java client instead of the Rest High Level Client for bulk requests.

This client should be compatible with both OpenSearch 1.x clusters and OpenSearch 2.x clusters. Additionally, it supports Java 8.

Originally, Data Prepper was going to update to use the Rest High Level Client version 2.0.0-rc1. However, that version requires Java 11.
@dlvenable dlvenable mentioned this issue Apr 29, 2022
Closed
4 tasks
@dlvenable dlvenable added this to the v1.4 milestone Apr 29, 2022
@dlvenable
Copy link
Member Author

Locally, I found that using the 2.0.0-rc1-SNAPSHOT opensearch-rest-high-level-client, Data Prepper's OpenSearchSinkIT test succeeds against: OpenSearch 2.0.0-rc1 (not the final RC1), 1.3.1, 1.2.4, 1.0.1, and OpenDistro 1.12.0. These are all the versions I've tested against and they all pass.

@dlvenable dlvenable self-assigned this May 2, 2022
@dlvenable
Copy link
Member Author

Currently, I expect that the 2.0.0-rc1 artifact will be available May 3.

opensearch-project/opensearch-build#1624 (comment)

@dlvenable dlvenable changed the title Update to use the latest OpenSearch 2.0.0 client (may be RC1) Update to use the opensearch-java client for _bulk requests May 9, 2022
dlvenable added a commit to dlvenable/data-prepper that referenced this issue May 12, 2022
@dlvenable
Migrated Data Prepper to use the opensearch-java client for bulk requ…
d57315b 
…ests rather than the REST High Level Client. opensearch-project#1347

Signed-off-by: David Venable <[email protected]>
@dlvenable dlvenable mentioned this issue May 12, 2022
Merged
4 tasks
dlvenable added a commit that referenced this issue May 13, 2022
@dlvenable
Migrated Data Prepper to use the opensearch-java client for bulk requ…
8b7d39c 
…ests rather than the REST High Level Client. #1347 (#1381)

Signed-off-by: David Venable <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlvenable dlvenable

Labels
None yet
Projects
Data Prepper Tracking Board
Archived in project
Milestone
v1.4
Development

Successfully merging a pull request may close this issue.

Migrated Data Prepper to use the opensearch-java client for bulk requests dlvenable/data-prepper
1 participant
@dlvenable