Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize read write lock constructs during translog upload to remote store #9636

Merged
merged 3 commits into from
Sep 18, 2023

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Aug 30, 2023

Description

Optimize read write lock constructs during translog upload to remote store

Related Issues

Resolves #10013

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 30, 2023

Compatibility status:

Checks if related components are compatible with change 5595fc2

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change 8c2604a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.classMethod

@codecov
Copy link

codecov bot commented Sep 12, 2023

Codecov Report

Merging #9636 (5595fc2) into main (e6dec29) will increase coverage by 0.12%.
Report is 9 commits behind head on main.
The diff coverage is 70.00%.

@@             Coverage Diff              @@
##               main    #9636      +/-   ##
============================================
+ Coverage     71.03%   71.16%   +0.12%     
- Complexity    58071    58139      +68     
============================================
  Files          4824     4824              
  Lines        273918   273928      +10     
  Branches      39918    39921       +3     
============================================
+ Hits         194591   194953     +362     
+ Misses        63017    62602     -415     
- Partials      16310    16373      +63     
Files Changed Coverage Δ
...g/transfer/TranslogCheckpointTransferSnapshot.java 79.22% <ø> (+1.01%) ⬆️
...rg/opensearch/index/translog/RemoteFsTranslog.java 70.56% <68.96%> (-0.52%) ⬇️
...dex/translog/transfer/TranslogTransferManager.java 73.51% <100.00%> (+0.50%) ⬆️

... and 443 files with indirect coverage changes

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@ashking94 ashking94 changed the title [Draft] Lockfix readlock Optimize read write lock constructs during translog upload to remote store Sep 13, 2023
@ashking94 ashking94 marked this pull request as ready for review September 13, 2023 06:25
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label Sep 18, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals}
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

@gbbafna gbbafna merged commit b5cc002 into opensearch-project:main Sep 18, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 18, 2023
…store (#9636)

Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit b5cc002)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@gbbafna gbbafna added the backport 2.10 Backport to 2.10 branch label Sep 18, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 18, 2023
…store (#9636)

Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit b5cc002)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gbbafna pushed a commit that referenced this pull request Sep 18, 2023
…store (#9636) (#10102)

(cherry picked from commit b5cc002)

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Sep 18, 2023
…store (#9636) (#10103)

(cherry picked from commit b5cc002)

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sarthakaggarwal97 pushed a commit to sarthakaggarwal97/OpenSearch that referenced this pull request Sep 20, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Sep 20, 2023
gbbafna pushed a commit that referenced this pull request Sep 22, 2023
…ore (#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 22, 2023
…ore (#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit cbff21d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Sep 22, 2023
…ore (opensearch-project#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
sarthakaggarwal97 pushed a commit to sarthakaggarwal97/OpenSearch that referenced this pull request Sep 24, 2023
…ore (opensearch-project#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
sachinpkale pushed a commit that referenced this pull request Sep 25, 2023
…ore (#10135) (#10187)

* Revert "Optimize read write lock constructs during translog upload to remote store (#9636)"

This reverts commit b5cc002.



* Downgrade writelock to readlock during translog upload



---------

Signed-off-by: Ashish Singh <[email protected]>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…ore (opensearch-project#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Ivan Brusic <[email protected]>
vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023
vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023
…ore (opensearch-project#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ore (opensearch-project#10135)

* Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)"

This reverts commit b5cc002.

Signed-off-by: Ashish Singh <[email protected]>

* Downgrade writelock to readlock during translog upload

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport 2.10 Backport to 2.10 branch enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Durability Issues and PRs related to the durability framework Storage Issues and PRs relating to data and metadata storage v2.11.0 Issues and PRs related to version 2.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize read write lock constructs during translog upload to remote store
3 participants