Revert uploading of remote cluster state manifest using min codec version #16403

soosinha · 2024-10-21T12:33:44Z

Description

When cluster is upgraded from 2.16 or lower to 2.17, the new cluster manager nodes are not able to join the cluster due to deserialization failure while downloading the manifest. The stacktrace is below:

FailedToCommitClusterStateException[publishing failed]; nested: XContentParseException[[-1:616] [cluster_metadata_manifest] failed to parse field [indices]]; nested: XContentParseException[[-1:599] [uploaded_index_metadata] unknown field [component_prefix]];
        at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1397)
        at org.opensearch.cluster.service.MasterService.publish(MasterService.java:384)
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:366)
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:228)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:215)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:257)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:891)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.opensearch.core.xcontent.XContentParseException: [-1:616] [cluster_metadata_manifest] failed to parse field [indices]
        at org.opensearch.core.xcontent.ObjectParser.parseValue(ObjectParser.java:594)
        at org.opensearch.core.xcontent.ObjectParser.parseArray(ObjectParser.java:585)
        at org.opensearch.core.xcontent.ObjectParser.parseSub(ObjectParser.java:620)
        at org.opensearch.core.xcontent.ObjectParser.parse(ObjectParser.java:356)
        at org.opensearch.core.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:188)
        at org.opensearch.gateway.remote.ClusterMetadataManifest.fromXContentV0(ClusterMetadataManifest.java:902)
        at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.deserialize(ChecksumBlobStoreFormat.java:163)
        at org.opensearch.gateway.remote.model.RemoteClusterMetadataManifest.deserialize(RemoteClusterMetadataManifest.java:137)
        at org.opensearch.gateway.remote.model.RemoteClusterMetadataManifest.deserialize(RemoteClusterMetadataManifest.java:32)
        at org.opensearch.common.remote.RemoteWriteableEntityBlobStore.read(RemoteWriteableEntityBlobStore.java:77)
        at org.opensearch.gateway.remote.RemoteManifestManager.fetchRemoteClusterMetadataManifest(RemoteManifestManager.java:240)
        at org.opensearch.gateway.remote.RemoteManifestManager.lambda$getLatestClusterMetadataManifest$3(RemoteManifestManager.java:194)
        at java.base/java.util.Optional.map(Optional.java:260)
        at org.opensearch.gateway.remote.RemoteManifestManager.getLatestClusterMetadataManifest(RemoteManifestManager.java:194)
        at org.opensearch.gateway.remote.RemoteClusterStateService.getLatestClusterMetadataManifest(RemoteClusterStateService.java:1045)
        at org.opensearch.gateway.GatewayMetaState$RemotePersistedState.setLastAcceptedState(GatewayMetaState.java:763)
        at org.opensearch.cluster.coordination.CoordinationState.handlePrePublish(CoordinationState.java:586)
        at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1392)
        ... 11 more
Caused by: org.opensearch.core.xcontent.XContentParseException: [-1:599] [uploaded_index_metadata] unknown field [component_prefix]
        at org.opensearch.core.xcontent.ObjectParser.lambda$errorOnUnknown$2(ObjectParser.java:129)

The above stack trace shows that the field component_prefix is present in the serialized entity but we do not recognize this field while deserializing. The field component_prefix was added under ClusterMetadataManifest.UploadedIndexMetadata in 2.15 version. In 2.17 version we added an enhancement to support version upgrade: #15216. As part of this change, we upload the manifest using the codec version corresponding to the min node version and download using the same codec version. But for UploadedIndexMetadata entity in the manifest we are using the latest codec version for uploading due to which the component_prefix field is always getting serialized which we are not able to deserialize.

Fix
Reverting the logic which uploads the manifest using the minimum codec version.

Related Issues

NA

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-10-21T13:13:21Z

❌ Gradle check result for 4ea63a5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-21T15:00:21Z

❌ Gradle check result for 850f6ae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-21T16:22:10Z

❌ Gradle check result for 2349d48: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-21T17:31:56Z

❌ Gradle check result for 5137b75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-23T10:48:29Z

❌ Gradle check result for 9835547: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-23T11:49:38Z

❌ Gradle check result for fdc4e15: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-23T13:00:27Z

✅ Gradle check result for e9288b4: SUCCESS

Bukhtawar

Sorry I am just catching up but based on my limited understanding we need to align codec version bumps with engine versions so that we can be sure we don't back to the older codec after a bump.
We also need to maintain all older codec versions for compatibility, we can remove that logic once there is a migration path of data on older codec versions.
While reading the file we read the codec version from the file name so that we don't need to guess what codec version was used for writing with the guarantee that writes were done in a version not greater than what the reader node can understand

soosinha · 2024-10-23T16:32:26Z

@Bukhtawar We are maintaining the older codec versions for reading reading data written using older codec. Codec version is already there in the filename to determine how to read it.

github-actions · 2024-10-23T17:17:13Z

❌ Gradle check result for bba8f04: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-23T18:46:06Z

❌ Gradle check result for d534f17: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sooraj Sinha <[email protected]>

github-actions · 2024-10-24T03:16:08Z

✅ Gradle check result for 4a6797c: SUCCESS

shwetathareja · 2024-10-25T05:37:12Z

@Bukhtawar Merging this PR. This only addresses manifest file and rest of the metadata is also written using latest codec version only. This may need more changes and thinking if we want to provide BWC for write operation as well.

Signed-off-by: Sooraj Sinha <[email protected]> (cherry picked from commit 4ad1be3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit 4ad1be3) Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

soosinha force-pushed the fix_uploaded_metadata branch from 4ea63a5 to 850f6ae Compare October 21, 2024 14:48

soosinha force-pushed the fix_uploaded_metadata branch from 850f6ae to 2349d48 Compare October 21, 2024 15:39

soosinha force-pushed the fix_uploaded_metadata branch from 2349d48 to 5137b75 Compare October 21, 2024 16:25

opensearch-ci-bot mentioned this pull request Oct 21, 2024

[AUTOCUT] Gradle Check Flaky Test Report for ApproximatePointRangeQueryTests #15807

Open

soosinha force-pushed the fix_uploaded_metadata branch from bd7249e to 44c19dd Compare October 23, 2024 08:27

soosinha changed the title ~~Serialize using codec version~~ Revert uploading of remote cluster state manifest using min codec version Oct 23, 2024

soosinha force-pushed the fix_uploaded_metadata branch from 44c19dd to 9835547 Compare October 23, 2024 10:12

soosinha force-pushed the fix_uploaded_metadata branch from 9835547 to fdc4e15 Compare October 23, 2024 10:56

soosinha force-pushed the fix_uploaded_metadata branch from fdc4e15 to e9288b4 Compare October 23, 2024 11:55

opensearch-ci-bot mentioned this pull request Oct 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

shwetathareja added skip-changelog bug Something isn't working backport 2.x Backport to 2.x branch labels Oct 23, 2024

shwetathareja approved these changes Oct 23, 2024

View reviewed changes

Bukhtawar reviewed Oct 23, 2024

View reviewed changes

soosinha force-pushed the fix_uploaded_metadata branch from bba8f04 to d534f17 Compare October 23, 2024 17:43

This was referenced Oct 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for ResourceAwareTasksTests #14293

Open

[AUTOCUT] Gradle Check Flaky Test Report for ReplicaToPrimaryPromotionIT #16160

Open

Revert uploading of manifest using min codec version

4a6797c

Signed-off-by: Sooraj Sinha <[email protected]>

soosinha force-pushed the fix_uploaded_metadata branch from d534f17 to 4a6797c Compare October 24, 2024 02:20

shwetathareja approved these changes Oct 25, 2024

View reviewed changes

shwetathareja merged commit 4ad1be3 into opensearch-project:main Oct 25, 2024
38 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 25, 2024

[Backport 2.x] Revert uploading of remote cluster state manifest using min codec version #16479

Merged

opensearch-ci-bot mentioned this pull request Oct 17, 2024

[AUTOCUT] Gradle Check Flaky Test Report for AllocationConstraintsTests #15831

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert uploading of remote cluster state manifest using min codec version #16403

Revert uploading of remote cluster state manifest using min codec version #16403

soosinha commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

Bukhtawar left a comment

soosinha commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 24, 2024

shwetathareja commented Oct 25, 2024

Revert uploading of remote cluster state manifest using min codec version #16403

Revert uploading of remote cluster state manifest using min codec version #16403

Conversation

soosinha commented Oct 21, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

Bukhtawar left a comment

Choose a reason for hiding this comment

soosinha commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 24, 2024

shwetathareja commented Oct 25, 2024

soosinha commented Oct 21, 2024 •

edited

Loading