Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(controller): datastore use read-write lock #2895

Conversation

goldenxinxing
Copy link
Contributor

@goldenxinxing goldenxinxing commented Oct 23, 2023

Description

Test case(use 200 thread to scan the same table which has 20w data)
image

Speed up the scan processing on multi scan request case.

Modules

  • UI
  • Controller
  • Agent
  • Client
  • Python-SDK
  • Others

Checklist

  • run code format and lint check
  • add unit test
  • add necessary doc

@goldenxinxing goldenxinxing added the enhancement ❤️ New feature or request label Oct 23, 2023
@goldenxinxing goldenxinxing force-pushed the chore/use_read_write_lock_in_datastore branch from 578422f to a131990 Compare October 23, 2023 08:52
@goldenxinxing goldenxinxing force-pushed the chore/use_read_write_lock_in_datastore branch from a131990 to d0baf45 Compare October 23, 2023 08:53
@codecov
Copy link

codecov bot commented Oct 23, 2023

Codecov Report

Merging #2895 (d0baf45) into main (1a716ae) will decrease coverage by 9.22%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #2895      +/-   ##
============================================
- Coverage     83.52%   74.31%   -9.22%     
- Complexity     3015     3018       +3     
============================================
  Files           542      417     -125     
  Lines         29765    14404   -15361     
  Branches       1724     1665      -59     
============================================
- Hits          24861    10704   -14157     
+ Misses         4137     2934    -1203     
+ Partials        767      766       -1     
Flag Coverage Δ
console ?
controller 74.31% <100.00%> (+0.04%) ⬆️
standalone ?
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...n/java/ai/starwhale/mlops/datastore/DataStore.java 85.53% <100.00%> (ø)
...tarwhale/mlops/datastore/impl/MemoryTableImpl.java 87.21% <100.00%> (+0.19%) ⬆️

... and 127 files with indirect coverage changes

@jialeicui jialeicui self-requested a review October 24, 2023 06:34
Copy link
Contributor

@jialeicui jialeicui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@goldenxinxing goldenxinxing merged commit 0728ec6 into star-whale:main Oct 25, 2023
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants