Add replication orchestration for a single shard #3533

Poojita-Raj · 2022-06-07T23:04:25Z

Description

This change implements the lifecycle of replication for a single shard in SegmentReplicationTarget (similar to RecoveryTarget).
It introduces 3 states as part of this lifecycle (INIT, REPLICATING and DONE).
We also introduced 5 tests in SegmentReplicationTargetTests that cater to the replication process (started by StartReplication).

This is a part of the process of merging our feature branch - feature/segment-replication - back into main by re-PRing our changes from the feature branch.
The breakdown of the plan to merge to main is detailed here: #2355
For added context on segment replication - here's the design proposal #2229

Issues Resolved

#3320

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-06-07T23:15:31Z

❌ Gradle Check failure 15d85d22506aac98326de25ffe13431f54bfbcf8
Log 5835

Reports 5835

mch2 · 2022-06-07T23:15:04Z

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

@@ -480,4 +481,55 @@ private TranslogDeletionPolicy getTranslogDeletionPolicy(EngineConfig engineConf
        );
    }

+    public synchronized void finalizeReplication(SegmentInfos infos, Store.MetadataSnapshot expectedMetadata, long seqNo)


The intention was for the segrep target to invoke the updateSegments method that exists already in this class when a replication event is finalizing, if there are changes can you pls add there? A lot of this looks like it was what we were doing in the poc inside InternalEngine and is not required here. I thin maybe the piece missing from the current method is the new call to store.cleanupAndVerify, though I think we can do all that work in the target instead?

mch2 · 2022-06-07T23:17:38Z

server/src/main/java/org/opensearch/index/engine/Engine.java

@@ -271,6 +271,8 @@ public void verifyEngineBeforeIndexClosing() throws IllegalStateException {
        }
    }

+    public void finalizeReplication(SegmentInfos infos, Store.MetadataSnapshot expectedMetadata, long seqNo) throws IOException {}


This method already exists within NRTEngine as updateSegments. I don't think we need to define it in Engine as we'll want to assert & cast the engine type when finalize is called on the shard.

mch2 · 2022-06-07T23:19:14Z

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

+        readerManager.maybeRefresh();
+    }
+
+    private void refreshLastCommittedSegmentInfos() {


I don't think this method is required, we already set the lastCommittedSegmentInfos inside of updateSegments with the passed in infos.

server/src/main/java/org/opensearch/index/shard/IndexShard.java

mch2 · 2022-06-07T23:21:55Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+     * Fetch a snapshot of the latest SegmentInfos held by the engine.
+     * @return {@link SegmentInfos}
+     */
+    public SegmentInfos getLatestSegmentInfos() {


This method should already exist getSegmentInfosSnapshot

mch2 · 2022-06-07T23:33:33Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

    }

    @Override
-    protected void onCancel(String reason) {
-        // TODO
+    public void onCancel(String reason) {


this function is declared in the superclass with the same logic, no need here?

mch2 · 2022-06-07T23:34:37Z

.../src/test/java/org/opensearch/index/seqno/SegmentReplicationLocalCheckpointTrackerTests.java

+
+import static org.hamcrest.Matchers.equalTo;
+
+public class SegmentReplicationLocalCheckpointTrackerTests extends OpenSearchTestCase {


I think these tests already exist in LocalCheckpointTrackerTests?

mch2 · 2022-06-07T23:36:58Z

server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java

+
+            @Override
+            public void onFailure(Exception e) {
+                assertEquals(exception, e.getCause().getCause());


ExceptionsHelper.unwrapCause(e) can help here.

mch2 · 2022-06-07T23:38:13Z

server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java

+        });
+    }
+
+    public void testFailure_finalizeReplication() throws IOException {


nit testFailure_finalizeReplicationThrows

server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java

mch2 · 2022-06-08T00:25:39Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationState.java

@@ -25,9 +25,11 @@ public class SegmentReplicationState implements ReplicationState {
     * @opensearch.internal
     */
    public enum Stage {
-        DONE((byte) 0),
+        INIT((byte) 1),


having DONE and INIT as 0 and 1 was intentional actually so new states can be added in a backward compatible way, see here. You should only need to add the new state REPLICATING((byte) 3) at the end.

opensearch-ci-bot · 2022-06-09T17:18:08Z

❌ Gradle Check failure 72360566597ab0968bac013bcaac62a5a7a94cf4
Log 5870

Reports 5870

reta · 2022-06-09T17:31:10Z

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

@@ -94,6 +95,7 @@ public synchronized void updateSegments(final SegmentInfos infos, long seqNo) th
            rollTranslogGeneration();
        }
        localCheckpointTracker.fastForwardProcessedSeqNo(seqNo);
+        readerManager.maybeRefresh();


Do you need to call maybeRefresh here? readerManager.updateSegments(infos) calls maybeRefresh() inside

opensearch-ci-bot · 2022-06-09T17:49:28Z

❌ Gradle Check failure c13007131ed46e311df2d36edf098deb7df625eb
Log 5871

Reports 5871

reta · 2022-06-09T17:59:13Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -1359,6 +1360,11 @@ public GatedCloseable<IndexCommit> acquireLastIndexCommit(boolean flushFirst) th
        }
    }

+    public void finalizeReplication(SegmentInfos infos, long seqNo) throws IOException {


It seems like we are adding the method to be called by everyone but only supported for NRTReplicationEngine. May we could open up the getEngine() and SegmentReplicationTarget could make the decision if updateSegments should be called?

For now I'm still concerned about opening up access of engine beyond IndexShard

Fair enouh, I think we can still do better than that:

public interface ReplicationEngine { void updateSegments(final SegmentInfos infos, long seqNo) throws IOException; }

Than IndexShard can do:

public Optional<ReplicationEngine> getReplicationEngine () { if (getEngine() instanceof ReplicationEngine)) { return Optional.of((ReplicationEngine)getEngine()); } else { return Optional.empty(); } }

The main idea here: if something is not implemented, it should not be called. In this case, if engine does not support replication - the getReplicationEngine() returns empty and the caller won't be able to make a call.

Wondering if we really need a new Engine interface, does the current Engine interface not perform anything that has to do with replication already?

@Bukhtawar the Engine is not exposed directly but performs everything needed, the goal here to expose only the small piece of engine functionality (ReplicationEngine) if the engine supports replication.

@reta I understand what the interface intends to do, what I have a concern on is the interface looks incomplete to me. I am not quite sure if void updateSegments(final SegmentInfos infos, long seqNo) throws IOException; is the only thing a ReplicationEngine should do

@Bukhtawar 👍

ReplicationEngine is a regular Engine implementation and right now the only additional functionality is updating & refreshing on an externally provided SegmentInfos in NRTReplicationEngine. @Bukhtawar what additional functionality do you think is missing? We could add a default implementation in Engine that throws a UnsupportedOperationException and only implement it in NRTReplicationEngine but I like the isolated interface.

I would revisit the Engine interface and see what can be extracted into a separate ReplicationEngine interface and decide on the split. Based on what you mention @mch2 we can also create a functional interface and call it a SegmentUpdateService instead so that it appears less overloaded and performs the limited responsibility it specialises in/is tasked with.

public interface SegmentUpdateService { @FunctionalInterface void updateSegments(final SegmentInfos infos, long seqNo) throws IOException; }

EDIT: @Poojita-Raj pointed me to #3533 (comment) for more context on this change. Given that the only caller of getReplicationEngine is the finalizeReplication method, could we mitigate this by simply making getReplicationEngine private, and then using NRTReplicationEngine directly as described below?

i agree that the current ReplicationEngine interface seems out of place and incomplete. Why not use NRTReplicationEngine directly?

public Optional<NRTReplicationEngine> getReplicationEngine () { if (getEngine() instanceof NRTReplicationEngine)) { return Optional.of((NRTReplicationEngine)getEngine()); } else { return Optional.empty(); } }

opensearch-ci-bot · 2022-06-09T18:06:55Z

❌ Gradle Check failure b6fcc87c17bff95ef8d02bb6d9d56c3124a59ed7
Log 5873

Reports 5873

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

mch2 · 2022-06-09T18:08:12Z

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

@@ -85,6 +85,7 @@ public NRTReplicationEngine(EngineConfig engineConfig) {

    public synchronized void updateSegments(final SegmentInfos infos, long seqNo) throws IOException {
        // Update the current infos reference on the Engine's reader.
+        assert engineConfig.isReadOnlyReplica() : "Only replicas should update Infos";


nit - This assert isn't really necessary, we assert the engine type in IndexShard & NRTReplicationEngine would never be wired up if it were not a readOnlyReplica.

opensearch-ci-bot · 2022-06-09T21:35:46Z

❌ Gradle Check failure bbf5df2a086b251b4d7575d438db968fdf038dd5
Log 5882

Reports 5882

opensearch-ci-bot · 2022-06-09T21:45:36Z

❌ Gradle Check failure 959b9eab352927aa4eceef0b6e1d8670a356db34
Log 5883

Reports 5883

Poojita-Raj · 2022-06-09T21:59:46Z

start gradle check

opensearch-ci-bot · 2022-06-09T22:27:43Z

❌ Gradle Check failure 959b9eab352927aa4eceef0b6e1d8670a356db34
Log 5884

Reports 5884

opensearch-ci-bot · 2022-06-09T22:52:13Z

❌ Gradle Check failure 959b9eab352927aa4eceef0b6e1d8670a356db34
Log 5886

Reports 5886

opensearch-ci-bot · 2022-06-10T00:04:30Z

❌ Gradle Check failure 426c7e513154779f15a7ba748f1a4a1e13f9ffe3
Log 5887

Reports 5887

Poojita-Raj · 2022-06-10T16:53:14Z

start gradle check

opensearch-ci-bot · 2022-06-10T17:15:59Z

❌ Gradle Check failure 426c7e513154779f15a7ba748f1a4a1e13f9ffe3
Log 5894

Reports 5894

opensearch-ci-bot · 2022-06-10T17:26:44Z

❌ Gradle Check failure 70ed463ac18a547fe8ae43b0ed4554f97d71d7d5
Log 5895

Reports 5895

Signed-off-by: Poojita Raj <[email protected]>

opensearch-ci-bot · 2022-06-13T20:57:02Z

✅ Gradle Check success ebb2b5e
Log 5946

Reports 5946

Bukhtawar · 2022-06-14T03:35:15Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+    public Optional<NRTReplicationEngine> getReplicationEngine() {
+        if (getEngine() instanceof NRTReplicationEngine) {


Why does IndexShard specifically need to know about NRTReplicationEngine can we simply return an Engine reference instead?

@Poojita-Raj we basically expose whole engine here which we were trying to avoid in the first place #3533 (comment)

@reta - Ah I see, I misunderstood. I've made the change to add the method updateSegments to Engine that by default throws an UnsupportedException, and which NRTReplicationEngine overrides to implement.

Thanks @Poojita-Raj , the interface (likeReplicationEngine) would solve this ambiguity: if the engine does not implement it - no instance is returned.

Thanks @reta, I've gone ahead with the interface implementation. Let me know if there are any other callouts :)

Bukhtawar · 2022-06-14T05:40:17Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationState.java

    public void setStage(Stage stage) {
-        this.stage = stage;
+        switch (stage) {
+            case INIT:
+                this.stage = Stage.INIT;
+                getIndex().reset();
+                break;
+            case REPLICATING:
+                validateAndSetStage(Stage.INIT, stage);
+                getIndex().start();
+                break;
+            case DONE:
+                validateAndSetStage(Stage.REPLICATING, stage);
+                getIndex().stop();
+                getTimer().stop();
+                break;
+            default:
+                throw new IllegalArgumentException("unknown SegmentReplicationState.Stage [" + stage + "]");
+        }
    }
 }


Can we have multiple threads invoking this, in which case we would need it to be synchronized.

Nope, it's only called by a single thread

Bukhtawar · 2022-06-14T05:43:32Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

+        finalizeListener.whenComplete(r -> listener.onResponse(null), listener::onFailure);
+    }
+
+    private void getFiles(CheckpointInfoResponse checkpointInfo, StepListener<GetSegmentFilesResponse> getFilesListener)


My main concern with these private methods are they probably aren't getting unit tested

While the private methods aren't being tested directly, the implementation details are being tested. For example - this part of the code checks if the files fetched are what we expected - https://github.com/opensearch-project/OpenSearch/blob/bd9bf6797719cadd66400626328f6b5c383c5d79/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java#L121-L123

Bukhtawar · 2022-06-14T05:47:38Z

server/src/main/java/org/opensearch/indices/replication/common/ReplicationFailedException.java

+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*
+ * Modifications Copyright OpenSearch Contributors. See
+ * GitHub history for details.
+ */
+


We probably can skip this part of the license on the new files

opensearch-ci-bot · 2022-06-14T23:04:58Z

✅ Gradle Check success bd9bf6797719cadd66400626328f6b5c383c5d79
Log 6009

Reports 6009

opensearch-ci-bot · 2022-06-15T19:41:02Z

✅ Gradle Check success 7787ad3cb8966c4c8edb1115778176ac6014cbc9
Log 6038

Reports 6038

mch2 · 2022-06-20T16:21:08Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

+        try {
+            multiFileWriter.close();
+        } finally {
+            // free store. increment happens in constructor


nit - this comment seems outdated.

server/src/main/java/org/opensearch/index/store/Store.java

kartg

Mostly nitpicks, but a few minor comments

server/src/main/java/org/opensearch/index/store/Store.java

server/src/main/java/org/opensearch/indices/replication/common/ReplicationFailedException.java

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

server/src/main/java/org/opensearch/OpenSearchException.java

kartg · 2022-06-20T22:46:13Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

+        );
+    }
+
+    Store.MetadataSnapshot getMetadataSnapshot() throws IOException {


Nitpick - add a comment as to why this isn't private

server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java

kartg · 2022-06-20T23:12:21Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -1359,6 +1360,11 @@ public GatedCloseable<IndexCommit> acquireLastIndexCommit(boolean flushFirst) th
        }
    }

+    public void finalizeReplication(SegmentInfos infos, long seqNo) throws IOException {


EDIT: @Poojita-Raj pointed me to #3533 (comment) for more context on this change. Given that the only caller of getReplicationEngine is the finalizeReplication method, could we mitigate this by simply making getReplicationEngine private, and then using NRTReplicationEngine directly as described below?

i agree that the current ReplicationEngine interface seems out of place and incomplete. Why not use NRTReplicationEngine directly?

public Optional<NRTReplicationEngine> getReplicationEngine () { if (getEngine() instanceof NRTReplicationEngine)) { return Optional.of((NRTReplicationEngine)getEngine()); } else { return Optional.empty(); } }

opensearch-ci-bot · 2022-06-21T05:23:55Z

✅ Gradle Check success 04832ad393b2f4877cbca9af352ad4739cb3c82e
Log 6170

Reports 6170

mch2 · 2022-06-21T17:44:49Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java

@@ -169,8 +167,7 @@ private void getFiles(CheckpointInfoResponse checkpointInfo, StepListener<GetSeg
                )
            );
        }
-        final List<StoreFileMetadata> filesToFetch = Stream.concat(diff.missing.stream(), diff.different.stream())
-            .collect(Collectors.toList());
+        final List<StoreFileMetadata> filesToFetch = diff.missing.stream().collect(Collectors.toList());


minor - diff.missing is already a list, so we shouldn't need to stream & collect to a new list.

opensearch-ci-bot · 2022-06-21T19:31:30Z

❌ Gradle Check failure 5af7fc9d976ec64a087696f42275b25ca3d883c3
Log 6190

Reports 6190

Signed-off-by: Poojita Raj <[email protected]>

opensearch-ci-bot · 2022-06-21T22:04:40Z

✅ Gradle Check success 531dedd
Log 6194

Reports 6194

mch2 · 2022-06-21T22:41:38Z

server/src/test/java/org/opensearch/index/store/StoreTests.java

+
+        // we want to ensure commitMetadata files are preserved after calling cleanup
+        for (String existingFile : store.directory().listAll()) {
+            assert (commitMetadata.contains(existingFile) == true);


nit - assertTrue(commitMetadata.contains(existingFile));

mch2

LGTM, thanks @Poojita-Raj

kartg

Thanks for patiently working through the feedback!

* Bump reactor-netty-core from 1.0.16 to 1.0.19 in /plugins/repository-azure (#3360) * Bump reactor-netty-core in /plugins/repository-azure Bumps [reactor-netty-core](https://github.com/reactor/reactor-netty) from 1.0.16 to 1.0.19. - [Release notes](https://github.com/reactor/reactor-netty/releases) - [Commits](reactor/reactor-netty@v1.0.16...v1.0.19) --- updated-dependencies: - dependency-name: io.projectreactor.netty:reactor-netty-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Type removal] _type removal from mocked responses of scroll hit tests (#3377) Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove _type deprecation from script and conditional processor (#3239) * [Type removal] Remove _type deprecation from script and conditional processor Signed-off-by: Suraj Singh <[email protected]> * Spotless check apply Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove _type from _bulk yaml test, scripts, unused constants (#3372) * [Type removal] Remove redundant _type deprecation checks in bulk request Signed-off-by: Suraj Singh <[email protected]> * [Type removal] bulk yaml tests validating deprecation on _type and removal from scripts Signed-off-by: Suraj Singh <[email protected]> * Fix Lucene-snapshots repo for jdk 17. (#3396) Signed-off-by: Marc Handalian <[email protected]> * Replace internal usages of 'master' term in 'server/src/internalClusterTest' directory (#2521) Signed-off-by: Tianli Feng <[email protected]> * [REMOVE] Cleanup deprecated thread pool types (FIXED_AUTO_QUEUE_SIZE) (#3369) Signed-off-by: Andriy Redko <[email protected]> * [Type removal] _type removal from tests of yaml tests (#3406) * [Type removal] _type removal from tests of yaml tests Signed-off-by: Suraj Singh <[email protected]> * Fix spotless failures Signed-off-by: Suraj Singh <[email protected]> * Fix assertion failures Signed-off-by: Suraj Singh <[email protected]> * Fix assertion failures in DoSectionTests Signed-off-by: Suraj Singh <[email protected]> * Add release notes for version 2.0.0 (#3410) Signed-off-by: Rabi Panda <[email protected]> * [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416) Upgrades to latest snapshot of lucene 9.2.0 in preparation for GA release. Signed-off-by: Nicholas Walter Knize <[email protected]> * Fix release notes for 2.0.0-rc1 version (#3418) This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases. Add back some missing type removal commits to the 2.0.0 release notes Signed-off-by: Rabi Panda <[email protected]> * Bump version 2.1 to Lucene 9.2 after upgrade (#3424) Bumps Version.V_2_1_0 lucene version to 9.2 after backporting upgrage. Signed-off-by: Nicholas Walter Knize <[email protected]> * Bump com.gradle.enterprise from 3.10 to 3.10.1 (#3425) Bumps com.gradle.enterprise from 3.10 to 3.10.1. --- updated-dependencies: - dependency-name: com.gradle.enterprise dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump reactor-core from 3.4.17 to 3.4.18 in /plugins/repository-azure (#3427) Bumps [reactor-core](https://github.com/reactor/reactor-core) from 3.4.17 to 3.4.18. - [Release notes](https://github.com/reactor/reactor-core/releases) - [Commits](reactor/reactor-core@v3.4.17...v3.4.18) --- updated-dependencies: - dependency-name: io.projectreactor:reactor-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (#3426) Bumps [gax-httpjson](https://github.com/googleapis/gax-java) from 0.101.0 to 0.103.1. - [Release notes](https://github.com/googleapis/gax-java/releases) - [Changelog](https://github.com/googleapis/gax-java/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/gax-java/commits) --- updated-dependencies: - dependency-name: com.google.api:gax-httpjson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [segment replication]Introducing common Replication interfaces for segment replication and recovery code paths (#3234) * RecoveryState inherits from ReplicationState + RecoveryTarget inherits from ReplicationTarget Signed-off-by: Poojita Raj <[email protected]> * Refactoring: mixedClusterVersion error fix + move Stage to ReplicationState Signed-off-by: Poojita Raj <[email protected]> * pull ReplicationListener into a top level class + add javadocs + address review comments Signed-off-by: Poojita Raj <[email protected]> * fix javadoc Signed-off-by: Poojita Raj <[email protected]> * review changes Signed-off-by: Poojita Raj <[email protected]> * Refactoring the hierarchy relationship between repl and recovery Signed-off-by: Poojita Raj <[email protected]> * style fix Signed-off-by: Poojita Raj <[email protected]> * move package common under replication Signed-off-by: Poojita Raj <[email protected]> * rename to replication Signed-off-by: Poojita Raj <[email protected]> * rename and doc changes Signed-off-by: Poojita Raj <[email protected]> * [Type removal] Remove type from BulkRequestParser (#3423) * [Type removal] Remove type handling in bulk request parser Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase Signed-off-by: Suraj Singh <[email protected]> * Adding CheckpointRefreshListener to trigger when Segment replication is turned on and Primary shard refreshes (#3108) * Intial PR adding classes and tests related to checkpoint publishing Signed-off-by: Rishikesh1159 <[email protected]> * Putting a Draft PR with all changes in classes. Testing is still not included in this commit. Signed-off-by: Rishikesh1159 <[email protected]> * Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs Signed-off-by: Rishikesh1159 <[email protected]> * Adding Unit test for checkpointRefreshListener Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Fixing import statements * Signed-off-by: Rishikesh1159 <[email protected]> * removing unused constructor in index shard Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments from last commit Signed-off-by: Rishikesh1159 <[email protected]> * Adding package-info.java files for two new packages Signed-off-by: Rishikesh1159 <[email protected]> * Adding test for null checkpoint publisher and addreesing PR comments Signed-off-by: Rishikesh1159 <[email protected]> * Add docs for indexshardtests and remove shard.refresh Signed-off-by: Rishikesh1159 <[email protected]> * Add a new Engine implementation for replicas with segment replication enabled. (#3240) * Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint. This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in Segment Replication. This is so that a Segrep Engine can match the logic of InternalEngine where the seqNo is incremented with each operation, but only persisted in the tracker on a flush. With Segment Replication we bump the processed number with each operation received index/delete/noOp, and invoke this method when we receive a new set of segments to bump the persisted seqNo. Signed-off-by: Marc Handalian <[email protected]> * Extract Translog specific engine methods into an abstract class. This change extracts translog specific methods to an abstract engine class so that other engine implementations can reuse translog logic. Signed-off-by: Marc Handalian <[email protected]> * Add a separate Engine implementation for replicas with segment replication enabled. This change adds a new engine intended to be used on replicas with segment replication enabled. This engine does not wire up an IndexWriter, but still writes all operations to a translog. The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless checks. Signed-off-by: Marc Handalian <[email protected]> * Fix :server:compileInternalClusterTestJava compilation. Signed-off-by: Marc Handalian <[email protected]> * Fix failing test naming convention check. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. - Removed isReadOnlyReplica from overloaded constructor and added feature flag checks. - Updated log msg in NRTReplicationReaderManager - cleaned up store ref counting in NRTReplicationEngine. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless check. Signed-off-by: Marc Handalian <[email protected]> * Remove TranslogAwareEngine and build translog in NRTReplicationEngine. Signed-off-by: Marc Handalian <[email protected]> * Fix formatting Signed-off-by: Marc Handalian <[email protected]> * Add missing translog methods to NRTEngine. Signed-off-by: Marc Handalian <[email protected]> * Remove persistent seqNo check from fastForwardProcessedSeqNo. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> * Add test specific to translog trimming. Signed-off-by: Marc Handalian <[email protected]> * Javadoc check. Signed-off-by: Marc Handalian <[email protected]> * Add failEngine calls to translog methods in NRTReplicationEngine. Roll xlog generation on replica when a new commit point is received. Signed-off-by: Marc Handalian <[email protected]> * Rename master to cluster_manager in the XContent Parser of ClusterHealthResponse (#3432) Signed-off-by: Tianli Feng <[email protected]> * Bump hadoop-minicluster in /test/fixtures/hdfs-fixture (#3359) Bumps hadoop-minicluster from 3.3.2 to 3.3.3. --- updated-dependencies: - dependency-name: org.apache.hadoop:hadoop-minicluster dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs (#3358) * Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs Bumps avro from 1.10.2 to 1.11.0. --- updated-dependencies: - dependency-name: org.apache.avro:avro dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix testSetAdditionalRolesCanAddDeprecatedMasterRole() by removing the initial assertion (#3441) Signed-off-by: Tianli Feng <[email protected]> * Replace internal usages of 'master' term in 'server/src/test' directory (#2520) * Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory. * Backwards compatibility is not impacted. * Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR #2424 Signed-off-by: Tianli Feng <[email protected]> * Removing unused method from TransportSearchAction (#3437) * Removing unused method from TransportSearchAction Signed-off-by: Ankit Jain <[email protected]> * Set term vector flags to false for ._index_prefix field (#1901). (#3119) * Set term vector flags to false for ._index_prefix field (#1901). Signed-off-by: Vesa Pehkonen <[email protected]> * Replaced the FieldType copy ctor with ctor for the prefix field and replaced setting the field type parameters with setIndexOptions(). (#1901) Signed-off-by: Vesa Pehkonen <[email protected]> * Added tests for term vectors. (#1901) Signed-off-by: Vesa Pehkonen <[email protected]> * Fixed code formatting error. Signed-off-by: Vesa Pehkonen <[email protected]> Co-authored-by: sdp <[email protected]> * [BUG] Fixing org.opensearch.monitor.os.OsProbeTests > testLogWarnCpuMessageOnlyOnes when cgroups are available but cgroup stats is not (#3448) Signed-off-by: Andriy Redko <[email protected]> * [Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events. (#3439) * Add SegmentReplicationTargetService to orchestrate replication events. This change introduces boilerplate classes for Segment Replication and a target service to orchestrate replication events. It also includes two refactors of peer recovery components for reuse. 1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into ReplicationTarget. 2. Extracts a component to execute retryable requests over the transport layer. Signed-off-by: Marc Handalian <[email protected]> * Code cleanup. Signed-off-by: Marc Handalian <[email protected]> * Make SegmentReplicationTargetService component final so that it can not be extended by plugins. Signed-off-by: Marc Handalian <[email protected]> * Bump azure-core-http-netty from 1.11.9 to 1.12.0 in /plugins/repository-azure (#3474) Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.11.9 to 1.12.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.11.9...azure-core_1.12.0) --- updated-dependencies: - dependency-name: com.azure:azure-core-http-netty dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update to Apache Lucene 9.2 (#3477) Signed-off-by: Andriy Redko <[email protected]> * Bump protobuf-java from 3.20.1 to 3.21.1 in /plugins/repository-hdfs (#3472) Signed-off-by: dependabot[bot] <[email protected]> * [Upgrade] Lucene-9.3.0-snapshot-823df23 (#3478) Upgrades to latest snapshot of lucene 9.3.0. Signed-off-by: Nicholas Walter Knize <[email protected]> * Filter out invalid URI and HTTP method in the error message of no handler found for a REST request (#3459) Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message. The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that. While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners. Signed-off-by: Tianli Feng <[email protected]> * Support use of IRSA for repository-s3 plugin credentials (#3475) * Support use of IRSA for repository-s3 plugin credentials Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Bump google-auth-library-oauth2-http from 0.20.0 to 1.7.0 in /plugins/repository-gcs (#3473) * Bump google-auth-library-oauth2-http in /plugins/repository-gcs Bumps google-auth-library-oauth2-http from 0.20.0 to 1.7.0. --- updated-dependencies: - dependency-name: com.google.auth:google-auth-library-oauth2-http dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Use variable to define the version of dependency google-auth-library-java Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Tianli Feng <[email protected]> * [Segment Replication] Added source-side classes for orchestrating replication events (#3470) This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard. Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass. A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation. Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets. Unit tests have been added/updated for all new functionality. Signed-off-by: Kartik Ganesh <[email protected]> * [Dependency upgrade] google-oauth-client to 1.33.3 (#3500) Signed-off-by: Suraj Singh <[email protected]> * move bash flag to set statement (#3494) Passing bash with flags to the first argument of /usr/bin/env requires its own flag to interpret it correctly. Rather than use `env -S` to split the argument, have the script `set -e` to enable the same behavior explicitly in preinst and postinst scripts. Also set `-o pipefail` for consistency. Closes: #3492 Signed-off-by: Cole White <[email protected]> * Support use of IRSA for repository-s3 plugin credentials: added YAML Rest test case (#3499) Signed-off-by: Andriy Redko <[email protected]> * Bump azure-storage-common from 12.15.0 to 12.16.0 in /plugins/repository-azure (#3517) * Bump azure-storage-common in /plugins/repository-azure Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.15.0 to 12.16.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.15.0...azure-storage-blob_12.16.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce (#3516) * Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.33.3 to 1.34.0. - [Release notes](https://github.com/googleapis/google-oauth-java-client/releases) - [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-oauth-java-client@v1.33.3...v1.34.0) --- updated-dependencies: - dependency-name: com.google.oauth-client:google-oauth-client dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix the support of RestClient Node Sniffer for version 2.x and update tests (#3487) Fix the support of RestClient Node Sniffer for OpenSearch 2.x, and update unit tests for OpenSearch. The current code contains the logic to be compatible with Elasticsearch 2.x version, which is conflict with OpenSearch 2.x, so removed that part of legacy code. * Update the script create_test_nodes_info.bash to dump the response of Nodes Info API GET _nodes/http for OpenSearch 1.0 and 2.0 version, which used for unit test. * Remove the support of Elasticsearch version 2.x for the Sniffer * Update unit test to validate the Sniffer compatible with OpenSearch 1.x and 2.x * Update the API response parser to meet the array notation (in ES 6.1 and above) for the node attributes setting. It will result the value of `node.attr` setting will not be parsed as array in the Sniffer, when using the Sniffer on cluster in Elasticsearch 6.0 and above. * Replace "master" node role with "cluster_manager" in unit test Signed-off-by: Tianli Feng <[email protected]> * Bump com.diffplug.spotless from 6.6.1 to 6.7.0 (#3513) Bumps com.diffplug.spotless from 6.6.1 to 6.7.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment (#3357) * Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment Bumps [guava](https://github.com/google/guava) from 18.0 to 23.0. - [Release notes](https://github.com/google/guava/releases) - [Commits](google/guava@v18.0...v23.0) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Add more ingorance of using internal java API sun.misc.Unsafe Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Tianli Feng <[email protected]> * Added bwc version 2.0.1 (#3452) Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> * Add release notes for 1.3.3 (#3549) Signed-off-by: Xue Zhou <[email protected]> * [Upgrade] Lucene-9.3.0-snapshot-b7231bb (#3537) Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted docs are not merged away in the test with the new mergeOnRefresh default policy. Signed-off-by: Nicholas Walter Knize <[email protected]> * [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Fixing VerifyVersionConstantsIT test failure (#3574) Signed-off-by: Andriy Redko <[email protected]> * Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic (#3571) * Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic Bumps [jettison](https://github.com/jettison-json/jettison) from 1.4.1 to 1.5.0. - [Release notes](https://github.com/jettison-json/jettison/releases) - [Commits](jettison-json/jettison@jettison-1.4.1...jettison-1.5.0) --- updated-dependencies: - dependency-name: org.codehaus.jettison:jettison dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1 in /plugins/repository-gcs (#3573) * Bump google-api-services-storage in /plugins/repository-gcs Bumps google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1. --- updated-dependencies: - dependency-name: com.google.apis:google-api-services-storage dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Upgrade Google HTTP Client to 1.42.0 Signed-off-by: Xue Zhou <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Xue Zhou <[email protected]> * Add flat_skew setting to node overload decider (#3563) * Add flat_skew setting to node overload decider Signed-off-by: Rishab Nahata <[email protected]> * Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment (#3572) * Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment Bumps xmlbeans from 5.0.3 to 5.1.0. --- updated-dependencies: - dependency-name: org.apache.xmlbeans:xmlbeans dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce (#3570) * Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.34.0 to 1.34.1. - [Release notes](https://github.com/googleapis/google-oauth-java-client/releases) - [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-oauth-java-client@v1.34.0...v1.34.1) --- updated-dependencies: - dependency-name: com.google.oauth-client:google-oauth-client dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider (#3428) * Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider Signed-off-by: Anshu Agarwal <[email protected]> * Added bwc version 1.3.4 (#3552) Signed-off-by: GitHub <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> * Support dynamic node role (#3436) * Support unknown node role Currently OpenSearch only supports several built-in nodes like data node role. If specify unknown node role, OpenSearch node will fail to start. This limit how to extend OpenSearch to support some extension function. For example, user may prefer to run ML tasks on some dedicated node which doesn't serve as any built-in node roles. So the ML tasks won't impact OpenSearch core function. This PR removed the limitation and user can specify any node role and OpenSearch will start node correctly with that unknown role. This opens the door for plugin developer to run specific tasks on dedicated nodes. Issue: #2877 Signed-off-by: Yaliang Wu <[email protected]> * fix cat nodes rest API spec Signed-off-by: Yaliang Wu <[email protected]> * fix mixed cluster IT failure Signed-off-by: Yaliang Wu <[email protected]> * add DynamicRole Signed-off-by: Yaliang Wu <[email protected]> * change generator method name Signed-off-by: Yaliang Wu <[email protected]> * fix failed docker test Signed-off-by: Yaliang Wu <[email protected]> * transform role name to lower case to avoid confusion Signed-off-by: Yaliang Wu <[email protected]> * transform the node role abbreviation to lower case Signed-off-by: Yaliang Wu <[email protected]> * fix checkstyle Signed-off-by: Yaliang Wu <[email protected]> * add test for case-insensitive role name change Signed-off-by: Yaliang Wu <[email protected]> * Rename package 'o.o.action.support.master' to 'o.o.action.support.clustermanager' (#3556) * Rename package org.opensearch.action.support.master to org.opensearch.action.support.clustermanager Signed-off-by: Tianli Feng <[email protected]> * Rename classes with master term in the package org.opensearch.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Deprecate classes in org.opensearch.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Remove pakcage o.o.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Move package-info back Signed-off-by: Tianli Feng <[email protected]> * Move package-info to new folder Signed-off-by: Tianli Feng <[email protected]> * Correct the package-info Signed-off-by: Tianli Feng <[email protected]> * Fixing flakiness of ShuffleForcedMergePolicyTests (#3591) Signed-off-by: Andriy Redko <[email protected]> * Deprecate classes in org.opensearch.action.support.master (#3593) Signed-off-by: Tianli Feng <[email protected]> * Add release notes for version 2.0.1 (#3595) Signed-off-by: Kunal Kotwani <[email protected]> * Fix NPE when minBound/maxBound is not set before being called. (#3605) Signed-off-by: George Apaaboah <[email protected]> * Added bwc version 2.0.2 (#3613) Co-authored-by: opensearch-ci-bot <[email protected]> * Fix false positive query timeouts due to using cached time (#3454) * Fix false positive query timeouts due to using cached time Signed-off-by: Ahmad AbuKhalil <[email protected]> * delegate nanoTime call to SearchContext Signed-off-by: Ahmad AbuKhalil <[email protected]> * add override to SearchContext getRelativeTimeInMillis to force non cached time Signed-off-by: Ahmad AbuKhalil <[email protected]> * Fix random gradle check failure issue 3584. (#3627) * [Segment Replication] Add components for segment replication to perform file copy. (#3525) * Add components for segment replication to perform file copy. This change adds the required components to SegmentReplicationSourceService to initiate copy and react to lifecycle events. Along with new components it refactors common file copy code from RecoverySourceHandler into reusable pieces. Signed-off-by: Marc Handalian <[email protected]> * Deprecate public methods and variables with master term in package 'org.opensearch.action.support.master' (#3617) Signed-off-by: Tianli Feng <[email protected]> * Add replication orchestration for a single shard (#3533) * implement segment replication target Signed-off-by: Poojita Raj <[email protected]> * test added Signed-off-by: Poojita Raj <[email protected]> * changes to tests + finalizeReplication Signed-off-by: Poojita Raj <[email protected]> * fix style check Signed-off-by: Poojita Raj <[email protected]> * addressing comments + fix gradle check Signed-off-by: Poojita Raj <[email protected]> * added test + addressed review comments Signed-off-by: Poojita Raj <[email protected]> * [BUG] opensearch crashes on closed client connection before search reply (#3626) * [BUG] opensearch crashes on closed client connection before search reply Signed-off-by: Andriy Redko <[email protected]> * Addressing code review comments Signed-off-by: Andriy Redko <[email protected]> * Add all deprecated method in the package with new name 'org.opensearch.action.support.clustermanager' (#3644) Signed-off-by: Tianli Feng <[email protected]> * Introduce TranslogManager implementations decoupled from the Engine (#3638) * Introduce decoupled translog manager interfaces Signed-off-by: Bukhtawar Khan <[email protected]> * Adding onNewCheckpoint to Start Replication on Replica Shard when Segment Replication is turned on (#3540) * Adding onNewCheckpoint and it's test to start replication. SCheck for latestcheckpoint and replaying logic is removed from this commit and will be added in a different PR Signed-off-by: Rishikesh1159 <[email protected]> * Changing binding/inject logic and addressing comments from PR Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Moving shouldProcessCheckpoint() to IndexShard, and removing some trace logs Signed-off-by: Rishikesh1159 <[email protected]> * applying spotlessApply Signed-off-by: Rishikesh1159 <[email protected]> * Adding more info to log statement in targetservice class Signed-off-by: Rishikesh1159 <[email protected]> * applying spotlessApply Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments on PR Signed-off-by: Rishikesh1159 <[email protected]> * Adding teardown() in SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <[email protected]> * fixing testShouldProcessCheckpoint() in SegmentReplicationTargetServiceTests Signed-off-by: Rishikesh1159 <[email protected]> * Removing CheckpointPublisherProvider in IndicesModule Signed-off-by: Rishikesh1159 <[email protected]> * spotless check apply Signed-off-by: Rishikesh1159 <[email protected]> * Remove class org.opensearch.action.support.master.AcknowledgedResponse (#3662) * Remove class org.opensearch.action.support.master.AcknowledgedResponse Signed-off-by: Tianli Feng <[email protected]> * Remove class org.opensearch.action.support.master.AcknowledgedRequest RequestBuilder ShardsAcknowledgedResponse Signed-off-by: Tianli Feng <[email protected]> * Restore AcknowledgedResponse and AcknowledgedRequest to package org.opensearch.action.support.master (#3669) Signed-off-by: Tianli Feng <[email protected]> * [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) (#3656) * [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) Signed-off-by: Andriy Redko <[email protected]> * Added test case for pluginZip with POM Signed-off-by: Andriy Redko <[email protected]> * Support both Gradle 6.8.x and Gradle 7.4.x Signed-off-by: Andriy Redko <[email protected]> * Adding 2.2.0 Bwc version to main (#3673) * Upgraded to t-digest 3.3. (#3634) * Revert renaming method onMaster() and offMaster() in interface LocalNodeMasterListener (#3686) Signed-off-by: Tianli Feng <[email protected]> * Upgrading AWS SDK dependency for native plugins (#3694) * Merge branch 'feature/point_in_time' of https://github.com/opensearch-project/OpenSearch into fb Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Suraj Singh <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Tianli Feng <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: Rabi Panda <[email protected]> Co-authored-by: Nick Knize <[email protected]> Co-authored-by: Poojita Raj <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Co-authored-by: Ankit Jain <[email protected]> Co-authored-by: vpehkone <[email protected]> Co-authored-by: sdp <[email protected]> Co-authored-by: Kartik Ganesh <[email protected]> Co-authored-by: Cole White <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Xue Zhou <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Xue Zhou <[email protected]> Co-authored-by: Rishab Nahata <[email protected]> Co-authored-by: Anshu Agarwal <[email protected]> Co-authored-by: Yaliang Wu <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> Co-authored-by: George Apaaboah <[email protected]> Co-authored-by: Ahmad AbuKhalil <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Sarat Vemulapalli <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

Signed-off-by: Rishikesh1159 <[email protected]>

…#3943 #3963 From main branch (#4181) * Resolving import conflict in Node.java and mergining PR #3525. Signed-off-by: Rishikesh1159 <[email protected]> * Resolving conflicts and merging PR #3533. Signed-off-by: Rishikesh1159 <[email protected]> * Resolving conflicts and Merging PR #3540. Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotlesscheck and fixing wildcard imports. Signed-off-by: Rishikesh1159 <[email protected]> * [Segment Replication] Fixing flaky test failure happening for testShardAlreadyReplicating() (#3943) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <[email protected]> * Fix possible flaky test for testBeforeIndexShardClosed_CancelsOngoingReplications() (#3963) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <[email protected]> * Removing assert segrep() in getProcessedLocalCheckpoint() of Index.shard class. Signed-off-by: Rishikesh1159 <[email protected]> * Adding back assert statement and make index setting to segment replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <[email protected]> * Revert "Adding back assert statement and make index setting to segment replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests." Signed-off-by: Rishikesh1159 <[email protected]> This reverts commit 8c5753b. Signed-off-by: Rishikesh1159 <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Poojita Raj <[email protected]>

mch2 reviewed Jun 7, 2022

View reviewed changes

mch2 reviewed Jun 8, 2022

View reviewed changes

Poojita-Raj changed the title ~~Segment reptarget~~ Add replication orchestration for a single shard Jun 9, 2022

Poojita-Raj marked this pull request as ready for review June 9, 2022 17:21

Poojita-Raj requested review from a team and reta as code owners June 9, 2022 17:21

reta reviewed Jun 9, 2022

View reviewed changes

Poojita-Raj force-pushed the segment-reptarget branch from c130071 to b6fcc87 Compare June 9, 2022 17:42

reta reviewed Jun 9, 2022

View reviewed changes

mch2 reviewed Jun 9, 2022

View reviewed changes

Poojita-Raj force-pushed the segment-reptarget branch from bbf5df2 to 959b9ea Compare June 9, 2022 21:39

Poojita-Raj force-pushed the segment-reptarget branch from 959b9ea to 426c7e5 Compare June 9, 2022 23:33

Poojita-Raj force-pushed the segment-reptarget branch from 426c7e5 to 70ed463 Compare June 10, 2022 17:05

Poojita-Raj added 4 commits June 13, 2022 08:43

implement segment replication target

f6cb092

Signed-off-by: Poojita Raj <[email protected]>

test added

704a109

Signed-off-by: Poojita Raj <[email protected]>

changes to tests + finalizeReplication

ed3fe28

Signed-off-by: Poojita Raj <[email protected]>

fix style check

f366918

Signed-off-by: Poojita Raj <[email protected]>

addressing comments + fix gradle check

ebb2b5e

Signed-off-by: Poojita Raj <[email protected]>

Poojita-Raj force-pushed the segment-reptarget branch from cb9100c to ebb2b5e Compare June 13, 2022 20:08

Poojita-Raj requested a review from reta June 13, 2022 21:39

Bukhtawar reviewed Jun 14, 2022

View reviewed changes

mch2 reviewed Jun 20, 2022

View reviewed changes

kartg reviewed Jun 20, 2022

View reviewed changes

mch2 reviewed Jun 21, 2022

View reviewed changes

Poojita-Raj force-pushed the segment-reptarget branch from 04832ad to 5af7fc9 Compare June 21, 2022 19:01

added test + addressed review comments

531dedd

Signed-off-by: Poojita Raj <[email protected]>

Poojita-Raj force-pushed the segment-reptarget branch from 5af7fc9 to 531dedd Compare June 21, 2022 21:29

mch2 reviewed Jun 21, 2022

View reviewed changes

mch2 approved these changes Jun 21, 2022

View reviewed changes

kartg approved these changes Jun 22, 2022

View reviewed changes

Poojita-Raj merged commit 3d4d5ca into opensearch-project:main Jun 22, 2022

Rishikesh1159 mentioned this pull request Aug 4, 2022

[Backport 2.x] [Segment Replication] Added source-side classes for orchestrating replication events. #4128

Merged

Rishikesh1159 pushed a commit to Rishikesh1159/OpenSearch that referenced this pull request Aug 9, 2022

Resolving conflicts and merging PR opensearch-project#3533.

6cffde4

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 mentioned this pull request Aug 9, 2022

[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

Merged

Rishikesh1159 pushed a commit to Rishikesh1159/OpenSearch that referenced this pull request Aug 10, 2022

Cherry-picking opensearch-project#3533.

ac0608d

Signed-off-by: Rishikesh1159 <[email protected]>


		import static org.hamcrest.Matchers.equalTo;

		public class SegmentReplicationLocalCheckpointTrackerTests extends OpenSearchTestCase {

		public Optional<NRTReplicationEngine> getReplicationEngine() {
		if (getEngine() instanceof NRTReplicationEngine) {

Add replication orchestration for a single shard #3533

Add replication orchestration for a single shard #3533

Conversation

Poojita-Raj commented Jun 7, 2022 • edited Loading

Description

Issues Resolved

Check List

opensearch-ci-bot commented Jun 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 9, 2022

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mch2 Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kartg Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 9, 2022

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 9, 2022

opensearch-ci-bot commented Jun 9, 2022

Poojita-Raj commented Jun 9, 2022 • edited Loading

opensearch-ci-bot commented Jun 9, 2022

opensearch-ci-bot commented Jun 9, 2022

opensearch-ci-bot commented Jun 10, 2022

Poojita-Raj commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 15, 2022

Choose a reason for hiding this comment

kartg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kartg Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 21, 2022

Choose a reason for hiding this comment

opensearch-ci-bot commented Jun 21, 2022

opensearch-ci-bot commented Jun 21, 2022

Choose a reason for hiding this comment

mch2 left a comment

Choose a reason for hiding this comment

kartg left a comment

Choose a reason for hiding this comment

Poojita-Raj commented Jun 7, 2022 •

edited

Loading

reta Jun 9, 2022 •

edited

Loading

reta Jun 14, 2022 •

edited

Loading

mch2 Jun 14, 2022 •

edited

Loading

kartg Jun 20, 2022 •

edited

Loading

Poojita-Raj commented Jun 9, 2022 •

edited

Loading

kartg Jun 20, 2022 •

edited

Loading