Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce sqlite database to store state information for CAS #1494

Merged
merged 4 commits into from
Dec 6, 2023

Conversation

jiangliu
Copy link
Collaborator

Relevant Issue (if applicable)

We are going to support runtime chunk deduplication, and this PR introduce sqlite database to store chunk state information for deduplication.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

@jiangliu jiangliu requested a review from a team as a code owner November 28, 2023 14:10
@jiangliu jiangliu requested review from bergwolf, liubin and gaius-qi and removed request for a team November 28, 2023 14:10
Copy link

codecov bot commented Nov 28, 2023

Codecov Report

Merging #1494 (5622645) into master (93ef71d) will increase coverage by 0.01%.
The diff coverage is 73.84%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1494      +/-   ##
==========================================
+ Coverage   62.76%   62.77%   +0.01%     
==========================================
  Files         123      125       +2     
  Lines       43340    43599     +259     
  Branches    43340    43599     +259     
==========================================
+ Hits        27202    27371     +169     
- Misses      14827    14873      +46     
- Partials     1311     1355      +44     
Files Coverage Δ
storage/src/cache/mod.rs 57.84% <ø> (ø)
storage/src/backend/localdisk.rs 48.10% <0.00%> (ø)
storage/src/cache/dedup/mod.rs 0.00% <0.00%> (ø)
storage/src/cache/dedup/db.rs 79.01% <79.01%> (ø)

... and 5 files with indirect coverage changes

@jiangliu jiangliu force-pushed the cas-db branch 5 times, most recently from 5b46804 to 7b490c4 Compare November 29, 2023 07:54
@jiangliu
Copy link
Collaborator Author

Simplified version of #1399, to support deduplication at runtime instead of build time.

@jiangliu jiangliu force-pushed the cas-db branch 5 times, most recently from 08b0d16 to 95c7832 Compare November 30, 2023 05:27
storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
Copy link
Contributor

@ccx1024cc ccx1024cc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A smoke test for code path is required.

storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
storage/src/cache/dedup/db.rs Outdated Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
storage/src/cache/dedup/db.rs Show resolved Hide resolved
storage/src/cache/mod.rs Show resolved Hide resolved
@jiangliu jiangliu force-pushed the cas-db branch 2 times, most recently from 465e31e to b8fc335 Compare December 2, 2023 11:59
xwb1136021767 and others added 3 commits December 4, 2023 10:33
Introduce sqlite to store CAS related information.

Signed-off-by: Huang Jianan <[email protected]>
Signed-off-by: xwb1136021767 <[email protected]>
Refine the sqlite database structure for storing CAS information.

Signed-off-by: Jiang Liu <[email protected]>
Move cas db from util into storage.

Signed-off-by: Jiang Liu <[email protected]>
Sqlite connection is not thread safe, so use connection pool to
support multi-threading.

Signed-off-by: Jiang Liu <[email protected]>
@ccx1024cc
Copy link
Contributor

A smoke test for code path is required.

Since this mod is basic module, which has no entry from user, smoke test is not required.

@ccx1024cc ccx1024cc self-requested a review December 6, 2023 07:42
@ccx1024cc
Copy link
Contributor

LGTM

@imeoer imeoer merged commit 50b8988 into dragonflyoss:master Dec 6, 2023
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants