Refactoring default and best_compression codec to their algorithm name #8745

sarthakaggarwal97 · 2023-07-18T11:10:27Z

Description

The aim of this PR is to refactor the default and best_compression codecs to their respective algorithm names.

Related Issues

Resolves #8695

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-18T11:39:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20401/
CommitID: 72f6bf0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-18T12:18:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/20405/
CommitID: dd094a0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-07-18T12:20:27Z

Codecov Report

Merging #8745 (98377e7) into main (723cab6) will decrease coverage by 0.04%.
The diff coverage is 76.00%.

@@             Coverage Diff              @@
##               main    #8745      +/-   ##
============================================
- Coverage     71.16%   71.13%   -0.04%     
- Complexity    57274    57275       +1     
============================================
  Files          4759     4759              
  Lines        269881   269904      +23     
  Branches      39488    39492       +4     
============================================
- Hits         192067   191998      -69     
- Misses        61616    61740     +124     
+ Partials      16198    16166      -32

Files Changed	Coverage Δ
.../index/codec/customcodecs/Lucene95CustomCodec.java	`76.92% <ø> (ø)`
...java/org/opensearch/index/engine/EngineConfig.java	`94.73% <68.42%> (-3.79%)`	⬇️
.../java/org/opensearch/index/codec/CodecService.java	`100.00% <100.00%> (ø)`
...opensearch/index/codec/customcodecs/ZstdCodec.java	`100.00% <100.00%> (ø)`
...arch/index/codec/customcodecs/ZstdNoDictCodec.java	`100.00% <100.00%> (ø)`

... and 454 files with indirect coverage changes

sarthakaggarwal97 · 2023-07-18T13:13:10Z

We would need to create issues for documentation and OSB changes

mgodwan · 2023-07-18T14:15:19Z

@sarthakaggarwal97 With this change, how will the upgrades work for clusters with existing indices? Won't we need to support the existing enum values?

sarthakaggarwal97 · 2023-07-18T14:37:53Z

@sarthakaggarwal97 With this change, how will the upgrades work for clusters with existing indices? Won't we need to support the existing enum values?

@mgodwan yes thanks for pointing out, it would be breaking for the cluster upgrade. Will re-add the support for the old enums.

github-actions · 2023-07-19T10:08:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20504/
CommitID: f7b3ba3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Poojita-Raj · 2023-07-19T20:32:59Z

@sarthakaggarwal97 With this change, how will the upgrades work for clusters with existing indices? Won't we need to support the existing enum values?

Agree with @mgodwan, we still need the support for "best_compression" and "default". A good follow up to this PR would be to make these changes in the documentation.

https://opensearch.org/docs/1.0/opensearch/rest-api/create-index/ - In the documentation, we can be transparent about the actual codec name of the codecs being used for best_compression and default.
Additionally, like we specify in the IllegalArgumentException for the index.codec setting, we should specify in the docs that we can provide one of these values: [default, best_compression, lz4, zlib, zstd, zstd_no_dict]

server/src/main/java/org/opensearch/index/codec/CodecService.java

server/src/main/java/org/opensearch/index/engine/EngineConfig.java

github-actions · 2023-07-20T10:24:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20582/
CommitID: 56d5686
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-20T12:01:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20588/
CommitID: a31fc0f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-20T12:39:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/20590/
CommitID: e8703c5
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-27T11:38:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21079/
CommitID: 53383ea
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-27T12:33:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21081/
CommitID: 94566f3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-27T13:22:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21087/
CommitID: d94d2f1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T00:46:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21196/
CommitID: 5a5c4f5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T05:35:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21223/
CommitID: 7bc4193
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Sarthak Aggarwal <[email protected]>

…ithm names Signed-off-by: Sarthak Aggarwal <[email protected]>

Signed-off-by: Sarthak Aggarwal <[email protected]>

github-actions · 2023-07-28T06:56:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/21233/
CommitID: 98377e7
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sarthakaggarwal97 changed the title ~~Refactor codec~~ Refactoring default and best_compression codec to their algorithm name Jul 18, 2023

sarthakaggarwal97 force-pushed the refactor-codec branch from 72f6bf0 to dd094a0 Compare July 18, 2023 11:45

Poojita-Raj reviewed Jul 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/codec/CodecService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/engine/EngineConfig.java Show resolved Hide resolved

sarthakaggarwal97 force-pushed the refactor-codec branch from f7b3ba3 to 56d5686 Compare July 20, 2023 10:03

sarthakaggarwal97 force-pushed the refactor-codec branch from 56d5686 to a31fc0f Compare July 20, 2023 11:39

sarthakaggarwal97 force-pushed the refactor-codec branch from a31fc0f to e8703c5 Compare July 20, 2023 12:06

linuxpi mentioned this pull request Jul 20, 2023

Remove remote translog settings to make all remote indices translog enabled #8719

Merged

6 tasks

sarthakaggarwal97 force-pushed the refactor-codec branch from e8703c5 to 53383ea Compare July 27, 2023 11:27

sarthakaggarwal97 force-pushed the refactor-codec branch from 53383ea to 94566f3 Compare July 27, 2023 12:03

sarthakaggarwal97 force-pushed the refactor-codec branch from 94566f3 to d94d2f1 Compare July 27, 2023 12:58

Poojita-Raj mentioned this pull request Jul 27, 2023

Amend tests to include all codecs #8907

Merged

sarthakaggarwal97 force-pushed the refactor-codec branch from d94d2f1 to 5a5c4f5 Compare July 28, 2023 00:08

sarthakaggarwal97 force-pushed the refactor-codec branch from 5a5c4f5 to 7bc4193 Compare July 28, 2023 04:58

Disallowing compression level for lz4 and best_compression codec

3fd0650

Signed-off-by: Sarthak Aggarwal <[email protected]>

sarthakaggarwal97 added 4 commits July 28, 2023 11:38

setting up codec settings interface for validation

e7bfc17

Signed-off-by: Sarthak Aggarwal <[email protected]>

added few tests

0efd22e

Signed-off-by: Sarthak Aggarwal <[email protected]>

Refactoring lz4 and best_compression codecs to their respective algor…

f288d78

…ithm names Signed-off-by: Sarthak Aggarwal <[email protected]>

reinstating default and best_compression

98377e7

Signed-off-by: Sarthak Aggarwal <[email protected]>

sarthakaggarwal97 force-pushed the refactor-codec branch from 7bc4193 to 98377e7 Compare July 28, 2023 06:10

sarthakaggarwal97 closed this Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring default and best_compression codec to their algorithm name #8745

Refactoring default and best_compression codec to their algorithm name #8745

sarthakaggarwal97 commented Jul 18, 2023 •

edited

Loading

github-actions bot commented Jul 18, 2023

github-actions bot commented Jul 18, 2023

codecov bot commented Jul 18, 2023 •

edited

Loading

sarthakaggarwal97 commented Jul 18, 2023

mgodwan commented Jul 18, 2023

sarthakaggarwal97 commented Jul 18, 2023

github-actions bot commented Jul 19, 2023

Poojita-Raj commented Jul 19, 2023

github-actions bot commented Jul 20, 2023

github-actions bot commented Jul 20, 2023

github-actions bot commented Jul 20, 2023

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

Refactoring default and best_compression codec to their algorithm name #8745

Refactoring default and best_compression codec to their algorithm name #8745

Conversation

sarthakaggarwal97 commented Jul 18, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jul 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 18, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jul 18, 2023 • edited Loading

Codecov Report

sarthakaggarwal97 commented Jul 18, 2023

mgodwan commented Jul 18, 2023

sarthakaggarwal97 commented Jul 18, 2023

github-actions bot commented Jul 19, 2023

Gradle Check (Jenkins) Run Completed with:

Poojita-Raj commented Jul 19, 2023

github-actions bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

sarthakaggarwal97 commented Jul 18, 2023 •

edited

Loading

codecov bot commented Jul 18, 2023 •

edited

Loading