Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: support small chunks mergence #884

Open
imeoer opened this issue Nov 23, 2022 · 3 comments
Open

proposal: support small chunks mergence #884

imeoer opened this issue Nov 23, 2022 · 3 comments
Labels

Comments

@imeoer
Copy link
Collaborator

imeoer commented Nov 23, 2022

image

Support merging small chunks (including the tailing chunk for big files) into one batch, so that we can compress and decompress the batch to improve the compression ratio, prefetch and io amplify performance.

@imeoer
Copy link
Collaborator Author

imeoer commented Nov 23, 2022

image

From @jiangliu: or consider using the BlobChunkInfoV2Ondisk struct.

@jiangliu
Copy link
Collaborator

We have built some infrastructure to support ZRan, which could also be used to support this feature requirement.

https://github.com/dragonflyoss/image-service/blob/master/storage/src/meta/chunk_info_v2.rs#L24

You can reuse or introduce a new struct like this one for ZRan:
https://github.com/dragonflyoss/image-service/blob/master/storage/src/meta/zran.rs#L16

@imeoer
Copy link
Collaborator Author

imeoer commented Nov 24, 2022

Related dragonfly issue: dragonflyoss/Dragonfly2#1858

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants