Make remote translog store path consistent to remote segment store #7947

linuxpi · 2023-06-07T04:59:39Z

Description

This PR updates remote translog store path to be consistent with segment.

Remote Segment Store Path

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/segments/data/
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/segments/metadata/

Previous Translog paths

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/metadata

New Translog paths

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/data/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/metadata

Related Issues

Addresses #1 in #3589 (comment)

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-07T05:24:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17006/
CommitID: f64d50d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-07T13:33:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17030/
CommitID: 670b02f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: bansvaru <[email protected]>

github-actions · 2023-06-07T15:46:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17040/
CommitID: 9773158
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-07T15:58:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

URL: https://build.ci.opensearch.org/job/gradle-check/17041/
CommitID: f8dbe09
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-07T16:07:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

URL: https://build.ci.opensearch.org/job/gradle-check/17042/
CommitID: 0b4640b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: bansvaru <[email protected]>

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

BhumikaSaini-Amazon

Thanks for adding this!

To confirm, would the new translog paths be:

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/translog/<shardid>/data/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/translog/<shardid>/metadata

OR

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/data/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/metadata

i.e. what is the sequence of <shardid> and translog?

github-actions · 2023-06-08T05:02:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17098/
CommitID: 42eb605
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: bansvaru <[email protected]>

linuxpi · 2023-06-08T07:01:00Z

Thanks for adding this!

To confirm, would the new translog paths be:

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/translog/<shardid>/data/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/translog/<shardid>/metadata

OR

s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/data/<primaryterm>/(*.tlog|*.ckp)
s3://remote-store-perf-test/cluster-1/zqQpla0QS_mFK3Q42FbQ0A/<shardid>/translog/metadata

i.e. what is the sequence of <shardid> and translog?

Thanks for catching this! I updated the description, i mistakenly put shardid after translog keyword

Signed-off-by: bansvaru <[email protected]>

github-actions · 2023-06-08T09:51:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/17109/
CommitID: a40649d

codecov · 2023-06-08T09:55:38Z

Codecov Report

Merging #7947 (c7e7d96) into main (8773509) will decrease coverage by 0.08%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #7947      +/-   ##
============================================
- Coverage     70.88%   70.80%   -0.08%     
- Complexity    56352    56390      +38     
============================================
  Files          4700     4700              
  Lines        266549   266549              
  Branches      39099    39099              
============================================
- Hits         188937   188742     -195     
- Misses        61632    61849     +217     
+ Partials      15980    15958      -22

Impacted Files	Coverage Δ
...rg/opensearch/index/translog/RemoteFsTranslog.java	`75.00% <100.00%> (ø)`
...dex/translog/transfer/TranslogTransferManager.java	`79.75% <100.00%> (-1.23%)`	⬇️

... and 454 files with indirect coverage changes

github-actions · 2023-06-08T10:08:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

URL: https://build.ci.opensearch.org/job/gradle-check/17111/
CommitID: c7e7d96
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…7947) Signed-off-by: bansvaru <[email protected]> (cherry picked from commit e6348c5) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]>

…7947) (#7964) (cherry picked from commit e6348c5) Signed-off-by: bansvaru <[email protected]>

…pensearch-project#7947) (opensearch-project#7964) (cherry picked from commit e6348c5) Signed-off-by: bansvaru <[email protected]>

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]> Signed-off-by: Rishab Nahata <[email protected]>

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

linuxpi force-pushed the remote-store-translog-segment-path-consistent branch from f64d50d to 670b02f Compare June 7, 2023 13:28

Make remote translog store path consistent to remote segment store

2ced9e2

Signed-off-by: bansvaru <[email protected]>

linuxpi force-pushed the remote-store-translog-segment-path-consistent branch from 670b02f to 9773158 Compare June 7, 2023 15:20

fix translog base path and update tests

0b4640b

Signed-off-by: bansvaru <[email protected]>

linuxpi force-pushed the remote-store-translog-segment-path-consistent branch from f8dbe09 to 0b4640b Compare June 7, 2023 15:30

Minor code refactoring

42eb605

Signed-off-by: bansvaru <[email protected]>

linuxpi marked this pull request as ready for review June 8, 2023 04:43

linuxpi requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja and dreamer-89 as code owners June 8, 2023 04:43

linuxpi requested review from tlfeng, VachaShah and dbwiddis as code owners June 8, 2023 04:43

gbbafna added the skip-changelog label Jun 8, 2023

gbbafna reviewed Jun 8, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java Outdated Show resolved Hide resolved

BhumikaSaini-Amazon reviewed Jun 8, 2023

View reviewed changes

fix spotless errors

a40649d

Signed-off-by: bansvaru <[email protected]>

rename instance method to reflect updated semantics

c7e7d96

Signed-off-by: bansvaru <[email protected]>

gbbafna approved these changes Jun 8, 2023

View reviewed changes

gbbafna merged commit e6348c5 into opensearch-project:main Jun 8, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Jun 8, 2023

opensearch-trigger-bot bot mentioned this pull request Jun 8, 2023

[Backport 2.x] Make remote translog store path consistent to remote segment store #7964

Merged

linuxpi deleted the remote-store-translog-segment-path-consistent branch June 8, 2023 11:32

dblock pushed a commit to dblock/OpenSearch that referenced this pull request Jun 8, 2023

Make remote translog store path consistent to remote segment store (o…

3ff3b69

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]>

sandeshkr419 pushed a commit to sandeshkr419/OpenSearch that referenced this pull request Jun 8, 2023

Make remote translog store path consistent to remote segment store (o…

e3bd089

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]>

gbbafna pushed a commit that referenced this pull request Jun 9, 2023

Make remote translog store path consistent to remote segment store (#…

6888bf0

…7947) (#7964) (cherry picked from commit e6348c5) Signed-off-by: bansvaru <[email protected]>

imRishN pushed a commit to imRishN/OpenSearch that referenced this pull request Jun 27, 2023

Make remote translog store path consistent to remote segment store (o…

d32a779

…pensearch-project#7947) Signed-off-by: bansvaru <[email protected]> Signed-off-by: Rishab Nahata <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make remote translog store path consistent to remote segment store #7947

Make remote translog store path consistent to remote segment store #7947

linuxpi commented Jun 7, 2023 •

edited

Loading

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 7, 2023

BhumikaSaini-Amazon left a comment

github-actions bot commented Jun 8, 2023

linuxpi commented Jun 8, 2023

github-actions bot commented Jun 8, 2023

codecov bot commented Jun 8, 2023 •

edited

Loading

github-actions bot commented Jun 8, 2023

Make remote translog store path consistent to remote segment store #7947

Make remote translog store path consistent to remote segment store #7947

Conversation

linuxpi commented Jun 7, 2023 • edited Loading

Description

Remote Segment Store Path

Previous Translog paths

New Translog paths

Related Issues

Check List

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 7, 2023

Gradle Check (Jenkins) Run Completed with:

BhumikaSaini-Amazon left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 8, 2023

Gradle Check (Jenkins) Run Completed with:

linuxpi commented Jun 8, 2023

github-actions bot commented Jun 8, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jun 8, 2023 • edited Loading

Codecov Report

github-actions bot commented Jun 8, 2023

Gradle Check (Jenkins) Run Completed with:

linuxpi commented Jun 7, 2023 •

edited

Loading

codecov bot commented Jun 8, 2023 •

edited

Loading