Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add filecache support in clear indices cache API #7498

Merged

Conversation

kotwanikunal
Copy link
Member

Description

Related Issues

Resolves #6030

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@kotwanikunal
Copy link
Member Author

@andrross / @reta I created a new PR since there were a bunch of changes.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod
      1 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown

@codecov-commenter
Copy link

codecov-commenter commented May 10, 2023

Codecov Report

Merging #7498 (c829953) into main (804bef4) will decrease coverage by 0.04%.
The diff coverage is 53.84%.

❗ Current head c829953 differs from pull request most recent head d85292e. Consider uploading reports for the commit d85292e to get more accurate results

@@             Coverage Diff              @@
##               main    #7498      +/-   ##
============================================
- Coverage     70.61%   70.57%   -0.04%     
+ Complexity    59794    59776      -18     
============================================
  Files          4896     4897       +1     
  Lines        286899   286805      -94     
  Branches      41350    41336      -14     
============================================
- Hits         202585   202405     -180     
- Misses        67633    67737     +104     
+ Partials      16681    16663      -18     
Impacted Files Coverage Δ
...s/cache/clear/ClearIndicesCacheRequestBuilder.java 16.66% <0.00%> (-3.34%) ⬇️
.../cache/clear/TransportClearIndicesCacheAction.java 27.27% <16.66%> (-3.98%) ⬇️
.../indices/cache/clear/ClearIndicesCacheRequest.java 55.26% <50.00%> (-1.41%) ⬇️
...earch/index/store/remote/utils/cache/LRUCache.java 91.87% <50.00%> (-0.54%) ⬇️
...rg/opensearch/client/IndicesRequestConverters.java 71.03% <100.00%> (+0.36%) ⬆️
...search/index/store/remote/filecache/FileCache.java 72.05% <100.00%> (+0.41%) ⬆️
...ndex/store/remote/utils/cache/RefCountedCache.java 100.00% <100.00%> (ø)
...index/store/remote/utils/cache/SegmentedCache.java 88.46% <100.00%> (+0.46%) ⬆️
...ion/admin/indices/RestClearIndicesCacheAction.java 60.00% <100.00%> (+2.85%) ⬆️

... and 502 files with indirect coverage changes

@kotwanikunal kotwanikunal force-pushed the file-cache-clear-api-v2 branch from b004da9 to 38dcb96 Compare May 10, 2023 04:43
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal kotwanikunal force-pushed the file-cache-clear-api-v2 branch from 38dcb96 to ab0f262 Compare May 10, 2023 16:06
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal kotwanikunal force-pushed the file-cache-clear-api-v2 branch from f4edc94 to c829953 Compare May 12, 2023 22:32
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal kotwanikunal force-pushed the file-cache-clear-api-v2 branch from c829953 to d88da42 Compare May 16, 2023 22:18
Copy link
Collaborator

@reta reta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please create a documentation issue [1] to update the existing Cache APIs with a new parameter, thank you.

[1] https://github.com/opensearch-project/documentation-website

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal
Copy link
Member Author

Gradle Check (Jenkins) Run Completed with:

java.lang.IllegalStateException: FileSystem Cache per segment capacity is less than single IndexInput default block size
	at __randomizedtesting.SeedInfo.seed([B61AEFA7CE988126:B1C1FF36A1C9CA84]:0)
	at org.opensearch.index.store.remote.filecache.FileCacheFactory.createFileCache(FileCacheFactory.java:57)
	at org.opensearch.index.store.remote.filecache.FileCacheFactory.createConcurrentLRUFileCache(FileCacheFactory.java:42)
	at 

Will update the cache size based on CI node specs.

@kotwanikunal
Copy link
Member Author

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal
Copy link
Member Author

Gradle Check (Jenkins) Run Completed with:

Unrelated to the changes. Created an issue - #7592

@kotwanikunal kotwanikunal added the backport 2.x Backport to 2.x branch label May 17, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

@kotwanikunal kotwanikunal merged commit a1e42b1 into opensearch-project:main May 17, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7498-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a1e42b1131d5d807dc66d13dd960ebe707da3f09
# Push it to GitHub
git push --set-upstream origin backport/backport-7498-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7498-to-2.x.

kotwanikunal added a commit to kotwanikunal/OpenSearch that referenced this pull request May 17, 2023
kotwanikunal added a commit to kotwanikunal/OpenSearch that referenced this pull request May 17, 2023
kotwanikunal added a commit to kotwanikunal/OpenSearch that referenced this pull request May 17, 2023
…7498)

Signed-off-by: Kunal Kotwani <[email protected]>
(cherry picked from commit a1e42b1)
Signed-off-by: Kunal Kotwani <[email protected]>
reta pushed a commit that referenced this pull request May 17, 2023
… filecache support in clear indices cache API (#7595)

* Remove LinkedDeque and replace with LinkedHashMap (#6968)

* Remove LinkedDeque and replace with LinkedHashMap

After the recent changes the usage of the LinkedDeque fits quite well
with the insertion order semantics of LinkedHashMap, which also allows
for constant time additions and removals.

Signed-off-by: Andrew Ross <[email protected]>

* Use class member reference now that lock is final

Signed-off-by: Andrew Ross <[email protected]>

---------

Signed-off-by: Andrew Ross <[email protected]>
(cherry picked from commit 65443ad)

* Add filecache support in clear indices cache API (#7498)

Signed-off-by: Kunal Kotwani <[email protected]>
(cherry picked from commit a1e42b1)
Signed-off-by: Kunal Kotwani <[email protected]>

---------

Signed-off-by: Kunal Kotwani <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
stephen-crawford pushed a commit to stephen-crawford/OpenSearch that referenced this pull request May 31, 2023
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Searchable Remote Index] Add a REST API for clearing file cache
5 participants