Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SegRep with Remote: Add Remote store as a segment replication source #7653

Merged
merged 3 commits into from
Jun 7, 2023

Conversation

ankitkala
Copy link
Member

@ankitkala ankitkala commented May 22, 2023

Description

Add remote store as a replication source for Segment Replication.

What's not covered in this PR (Fast follows):

  • Fixing the failing integration tests around pressureServiceStats and scrollQuery.
  • Refactoring required in SegmentReplicationTarget & SegmentReplicationSource for better alignment with remote store.
  • Changes in remote store to upload lucene version as part of metadata. This is getting used for reconstructing StoreFileMetadata in getCheckpointMetadata. This will depend on how we do the refactor mentioned above.

Related Issues

Resolves #4793

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@ankitkala ankitkala changed the title Add Remote store as a segment replication source SegRep with Remote: Add Remote store as a segment replication source May 22, 2023
@ankitkala ankitkala marked this pull request as ready for review May 22, 2023 11:22
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@sachinpkale
Copy link
Member

Overall changes look good. Please fix the build.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@ankitkala ankitkala requested a review from dbwiddis as a code owner May 24, 2023 08:54
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@ankitkala ankitkala force-pushed the remote_upload_hook_source branch from f1da45c to 35ae04a Compare June 7, 2023 09:19
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <[email protected]>
@ankitkala ankitkala force-pushed the remote_upload_hook_source branch from e3c48fe to 54c820d Compare June 7, 2023 11:14
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

@mch2 mch2 merged commit 2a3683e into opensearch-project:main Jun 7, 2023
@mch2 mch2 added the backport 2.x Backport to 2.x branch label Jun 7, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7653-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2a3683e4007de25fd1304912bc7405b07552991a
# Push it to GitHub
git push --set-upstream origin backport/backport-7653-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7653-to-2.x.

stephen-crawford pushed a commit to stephen-crawford/OpenSearch that referenced this pull request Jun 7, 2023
…pensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
reta added a commit that referenced this pull request Jun 8, 2023
…1.0.83 (#7933)

* Update network schema

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Add joni and jcoding to path

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update Apache Lucene to 9.7.0-snapshot-204acc3 (#7887)

Signed-off-by: Andriy Redko <[email protected]>

* Add bwc version 2.8.1 (#7942)

Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

* Fix changelog entries (#7946)

Move entry for #7321 to correct section and fix the malformed entry
from #7452.

Signed-off-by: Andrew Ross <[email protected]>

* SegRep with Remote: Add Remote store as a segment replication source (#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>

* rebase

Signed-off-by: Stephen Crawford <[email protected]>

* remove empty line

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Ankit Kala <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Ankit Kala <[email protected]>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request Jun 8, 2023
…pensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
dblock pushed a commit to dblock/OpenSearch that referenced this pull request Jun 8, 2023
…pensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
dblock pushed a commit to dblock/OpenSearch that referenced this pull request Jun 8, 2023
…1.0.83 (opensearch-project#7933)

* Update network schema

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Add joni and jcoding to path

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887)

Signed-off-by: Andriy Redko <[email protected]>

* Add bwc version 2.8.1 (opensearch-project#7942)

Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

* Fix changelog entries (opensearch-project#7946)

Move entry for opensearch-project#7321 to correct section and fix the malformed entry
from opensearch-project#7452.

Signed-off-by: Andrew Ross <[email protected]>

* SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>

* rebase

Signed-off-by: Stephen Crawford <[email protected]>

* remove empty line

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Ankit Kala <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Ankit Kala <[email protected]>
sandeshkr419 pushed a commit to sandeshkr419/OpenSearch that referenced this pull request Jun 8, 2023
…pensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
sandeshkr419 pushed a commit to sandeshkr419/OpenSearch that referenced this pull request Jun 8, 2023
…1.0.83 (opensearch-project#7933)

* Update network schema

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Add joni and jcoding to path

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887)

Signed-off-by: Andriy Redko <[email protected]>

* Add bwc version 2.8.1 (opensearch-project#7942)

Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

* Fix changelog entries (opensearch-project#7946)

Move entry for opensearch-project#7321 to correct section and fix the malformed entry
from opensearch-project#7452.

Signed-off-by: Andrew Ross <[email protected]>

* SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>

* rebase

Signed-off-by: Stephen Crawford <[email protected]>

* remove empty line

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Ankit Kala <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Ankit Kala <[email protected]>
gbbafna pushed a commit that referenced this pull request Jun 12, 2023
imRishN pushed a commit to imRishN/OpenSearch that referenced this pull request Jun 27, 2023
…1.0.83 (opensearch-project#7933)

* Update network schema

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Add joni and jcoding to path

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887)

Signed-off-by: Andriy Redko <[email protected]>

* Add bwc version 2.8.1 (opensearch-project#7942)

Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

* Fix changelog entries (opensearch-project#7946)

Move entry for opensearch-project#7321 to correct section and fix the malformed entry
from opensearch-project#7452.

Signed-off-by: Andrew Ross <[email protected]>

* SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>

* rebase

Signed-off-by: Stephen Crawford <[email protected]>

* remove empty line

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Ankit Kala <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Ankit Kala <[email protected]>
Signed-off-by: Rishab Nahata <[email protected]>
@@ -6637,6 +6637,7 @@ public void testLastRefreshCheckpoint() throws Exception {
while (done.get() == false) {
long checkPointBeforeRefresh = engine.getProcessedLocalCheckpoint();
engine.refresh("test", randomFrom(Engine.SearcherScope.values()), true);
assertThat(engine.currentOngoingRefreshCheckpoint(), greaterThanOrEqualTo(engine.lastRefreshedCheckpoint()));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ankitkala this assertion seems to be flaky (#9029), could you please take a look? thank you

shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…1.0.83 (opensearch-project#7933)

* Update network schema

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

* Add joni and jcoding to path

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json

Signed-off-by: Stephen Crawford <[email protected]>

* Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887)

Signed-off-by: Andriy Redko <[email protected]>

* Add bwc version 2.8.1 (opensearch-project#7942)

Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

* Fix changelog entries (opensearch-project#7946)

Move entry for opensearch-project#7321 to correct section and fix the malformed entry
from opensearch-project#7452.

Signed-off-by: Andrew Ross <[email protected]>

* SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653)

* SegRep with Remote: Add Remote store as a segment replication source

Signed-off-by: Ankit Kala <[email protected]>

* Fix Gradle check

Signed-off-by: Ankit Kala <[email protected]>

* Retrying Gradle check

Signed-off-by: Ankit Kala <[email protected]>

---------

Signed-off-by: Ankit Kala <[email protected]>

* rebase

Signed-off-by: Stephen Crawford <[email protected]>

* remove empty line

Signed-off-by: Stephen Crawford <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Ankit Kala <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Ankit Kala <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add segment replication source implementation for remote store integration
6 participants