[Segment Replication] Support realtime reads for GET requests #9212

Poojita-Raj · 2023-08-09T23:30:12Z

Description

This change supports realtime reads for GET requests when segment replication is enabled.
If there are no preferences mentioned for the incoming GET request then we route it to primary shards which serve strong reads by parsing source from the translog.

Related Issues

Resolves #8536

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/src/main/java/org/opensearch/action/get/TransportGetAction.java

...er/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java

github-actions · 2023-08-10T00:06:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22294/
CommitID: 55645ac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-10T00:13:43Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 36m 47s

github-actions · 2023-08-10T00:16:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22295/
CommitID: d3413fb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/get/TransportGetAction.java

github-actions · 2023-08-16T20:26:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22934/
CommitID: ca23136
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-16T20:33:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22935/
CommitID: 810ad17
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-16T20:49:52Z

Compatibility status:

Checks if related components are compatible with change 7ffcd65

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

opensearch-trigger-bot · 2023-08-16T20:53:05Z

Compatibility status:

Checks if related components are compatible with change 7ffcd65

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

server/src/main/java/org/opensearch/action/get/TransportGetAction.java

...er/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java

server/src/test/java/org/opensearch/action/get/TransportGetActionTests.java

opensearch-trigger-bot · 2023-08-16T21:32:02Z

Compatibility status:

Checks if related components are compatible with change 7ffcd65

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-16T21:46:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22940/
CommitID: 879709a

Signed-off-by: Poojita Raj <[email protected]>

server/src/test/java/org/opensearch/action/get/TransportGetActionTests.java

server/src/main/java/org/opensearch/action/get/TransportGetAction.java

github-actions · 2023-08-16T23:11:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22943/
CommitID: 9fece7c

Signed-off-by: Poojita Raj <[email protected]>

github-actions · 2023-08-17T17:54:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.CloneSnapshotIT.testCloneShallowSnapshotIndex

URL: https://build.ci.opensearch.org/job/gradle-check/22985/
CommitID: 49d6ebf
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-08-17T20:30:17Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9212-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 914544b182744d7c523f5d1f4f82ab7df256f424
# Push it to GitHub
git push --set-upstream origin backport/backport-9212-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9212-to-2.x.

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> (cherry picked from commit 914544b)

…#9421)

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]>

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Gagan Juneja <[email protected]>

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Kiran Reddy <[email protected]>

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…arch-project#9212) * route to pri shards Signed-off-by: Poojita Raj <[email protected]> * Add changelog entry Signed-off-by: Poojita Raj <[email protected]> * refactor + add multiGet support Signed-off-by: Poojita Raj <[email protected]> * address changes Signed-off-by: Poojita Raj <[email protected]> * add comment and rebase Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

mch2 reviewed Aug 9, 2023

View reviewed changes

Bukhtawar reviewed Aug 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/get/TransportGetAction.java Outdated Show resolved Hide resolved

dreamer-89 reviewed Aug 11, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/get/TransportGetAction.java Outdated Show resolved Hide resolved

Poojita-Raj force-pushed the realtimeGet branch from 01e4f2e to c313113 Compare August 11, 2023 00:33

mch2 reviewed Aug 16, 2023

View reviewed changes

add comment and rebase

9fece7c

Signed-off-by: Poojita Raj <[email protected]>

Poojita-Raj force-pushed the realtimeGet branch from 879709a to 9fece7c Compare August 16, 2023 22:26

andrross reviewed Aug 16, 2023

View reviewed changes

address review comments

49d6ebf

Signed-off-by: Poojita Raj <[email protected]>

mch2 approved these changes Aug 17, 2023

View reviewed changes

Rishikesh1159 approved these changes Aug 17, 2023

View reviewed changes

mch2 merged commit 914544b into opensearch-project:main Aug 17, 2023
10 of 11 checks passed

mch2 added the backport 2.x Backport to 2.x branch label Aug 17, 2023

opensearch-trigger-bot bot added the backport-failed label Aug 17, 2023

Poojita-Raj mentioned this pull request Aug 17, 2023

[Backport 2.x][Segment Replication] Support realtime reads for GET requests (#9212) #9421

Merged

mch2 pushed a commit that referenced this pull request Aug 18, 2023

[Segment Replication] Support realtime reads for GET requests (#9212) (…

2b6fa05

…#9421)

Poojita-Raj mentioned this pull request Sep 7, 2023

[DOC]Segment replication added consideration opensearch-project/documentation-website#4984

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Support realtime reads for GET requests #9212

[Segment Replication] Support realtime reads for GET requests #9212

Poojita-Raj commented Aug 9, 2023 •

edited

Loading

github-actions bot commented Aug 10, 2023

opensearch-trigger-bot bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

opensearch-trigger-bot bot commented Aug 16, 2023

opensearch-trigger-bot bot commented Aug 16, 2023

opensearch-trigger-bot bot commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

github-actions bot commented Aug 17, 2023

opensearch-trigger-bot bot commented Aug 17, 2023

[Segment Replication] Support realtime reads for GET requests #9212

[Segment Replication] Support realtime reads for GET requests #9212

Conversation

Poojita-Raj commented Aug 9, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 16, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

opensearch-trigger-bot bot commented Aug 16, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

opensearch-trigger-bot bot commented Aug 16, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 17, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 17, 2023

Poojita-Raj commented Aug 9, 2023 •

edited

Loading