Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove remote translog settings to make all remote indices translog enabled #8719

Merged

Conversation

linuxpi
Copy link
Collaborator

@linuxpi linuxpi commented Jul 17, 2023

Description

  • Removes the option to create remote indices without translog remote store enabled. All remote indices created should be with segment and translog remote store.
  • Also, refactors segment store settings to move it in a nested structure under "segment" key

Sample json to create remote translog index:

Before this change
{               
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "replication": {
        "type": "SEGMENT"
      },
      "remote_store": {
        "enabled": true,
        "repository": "my-fs-repository", "translog": {"enabled": true, "repository": "my-fs-repository", "buffer_interval": "650ms"}
      }
    }
  }
}
After this change
{               
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "replication": {
        "type": "SEGMENT"
      },
      "remote_store": {
        "enabled": true,
        "segment": {"repository": "my-segment-repository"},
        "translog": {"repository": "my-translog-repository",  "buffer_interval": "650ms"}
      }
    }
  }
}
  • A lot of tests were updated which just relied on remote segment or were using remote translog settings, leading to large no files being touched.
  • Please do take extra care while reviewing server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java changes.

Related Issues

Resolves #8769

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 1edc46c to f281cb9 Compare July 17, 2023 12:18
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from f281cb9 to 31a3104 Compare July 17, 2023 13:36
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 31a3104 to 4999d3e Compare July 18, 2023 09:35
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch 2 times, most recently from 3b1f449 to 7f77b5a Compare July 18, 2023 18:37
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 7f77b5a to 01809a0 Compare July 19, 2023 04:45
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 01809a0 to 4fbf5dc Compare July 19, 2023 04:54
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 9b60878 to ed153bb Compare July 19, 2023 09:19
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch 2 times, most recently from 92eefff to 08441f3 Compare July 19, 2023 10:04
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch 2 times, most recently from 4d3d7a0 to 98c77fc Compare July 19, 2023 10:56
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch 2 times, most recently from 48ab547 to 7bac123 Compare July 19, 2023 12:45
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi
Copy link
Collaborator Author

linuxpi commented Jul 20, 2023

Flaky tests:

Failing in other builds as well

   [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_2/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_3/)
    [org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush_4/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_2/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_3/)
    [org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore/SegmentReplicationUsingRemoteStoreIT/testReplicaHasDiffFilesThanPrimary_4/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_2/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_3/)
    [org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/20586/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush_4/)

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from fadbdcf to 486964b Compare July 20, 2023 15:19
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

@codecov
Copy link

codecov bot commented Jul 20, 2023

Codecov Report

Merging #8719 (954a5fa) into main (6cc8da5) will increase coverage by 0.28%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #8719      +/-   ##
============================================
+ Coverage     70.61%   70.89%   +0.28%     
- Complexity    56909    57148     +239     
============================================
  Files          4771     4771              
  Lines        270253   270235      -18     
  Branches      39502    39497       -5     
============================================
+ Hits         190837   191594     +757     
+ Misses        63250    62540     -710     
+ Partials      16166    16101      -65     
Impacted Files Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java 93.18% <ø> (ø)
...pensearch/common/settings/IndexScopedSettings.java 100.00% <ø> (ø)
...in/java/org/opensearch/indices/IndicesService.java 70.86% <ø> (+1.31%) ⬆️
...org/opensearch/cluster/metadata/IndexMetadata.java 86.16% <100.00%> (+0.46%) ⬆️
...h/cluster/metadata/MetadataCreateIndexService.java 80.07% <100.00%> (+0.24%) ⬆️
.../main/java/org/opensearch/index/IndexSettings.java 85.10% <100.00%> (-0.04%) ⬇️

... and 497 files with indirect coverage changes

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 486964b to ef7f650 Compare July 20, 2023 18:41
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicationAfterForceMerge

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from ef7f650 to 2ab9c84 Compare July 21, 2023 04:22
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the remote-store-setting-refactor branch from 2ab9c84 to 954a5fa Compare July 21, 2023 04:50
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

@sachinpkale sachinpkale merged commit c47ec1b into opensearch-project:main Jul 21, 2023
9 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Jul 21, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8719-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c47ec1ba79757ff3fdb7ac59c69209c0dcaa3139
# Push it to GitHub
git push --set-upstream origin backport/backport-8719-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8719-to-2.x.

@sachinpkale sachinpkale added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jul 21, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8719-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c47ec1ba79757ff3fdb7ac59c69209c0dcaa3139
# Push it to GitHub
git push --set-upstream origin backport/backport-8719-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8719-to-2.x.

linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request Jul 21, 2023
…nabled (opensearch-project#8719)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
(cherry picked from commit c47ec1b)
sachinpkale pushed a commit that referenced this pull request Jul 22, 2023
…nabled (#8719) (#8813)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
(cherry picked from commit c47ec1b)

Signed-off-by: Varun Bansal <[email protected]>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
…nabled (opensearch-project#8719)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…nabled (opensearch-project#8719)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
Signed-off-by: Kaushal Kumar <[email protected]>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…nabled (opensearch-project#8719)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
Signed-off-by: Ivan Brusic <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…nabled (opensearch-project#8719)

Remove remote translog setting and restrict to remote only index creation

---------

Signed-off-by: bansvaru <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Remote Store] All remote indices will have both translog and segment backed remote store.
3 participants