support generating new image with separate blob for prefetched files #1649
+826
−36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Major changes: 1. Added compatibility for rafs v5/v6 formats. 2. Set IS_PREFETCHBLOB flag in BlobInfo for prefetchblob. 3. Introduced two new options for storing prefetchblobID path and the optimized bootstrap path. Signed-off-by: Xing Ma <[email protected]>
Maxing1997
requested review from
imeoer,
jiangliu and
gaius-qi
and removed request for
a team
Maxing1997
changed the title
We can statically analyze the image entrypoint dependency, or use runtime dynamic analysis technologies such as ebpf, fanotify, metric, etc. to obtain the container file access pattern, and then build this part of data into an independent image layer: * preferentially fetch blob during the image startup phase to reduce network and disk IO. * avoid frequent image builds, allows for better local cache utilization. Implement optimize subcommand of nydusify to generate a new image, which references a new blob included prefetch file chunks. ``` nydusify optimize --policy separated-prefetch-blob \ --source $existed-nydus-image \ --target $new-nydus-image \ --prefetch-files /path/to/prefetch-files ``` More detailed process is as follows: 1. nydusify first downloads the source image and bootstrap, utilize nydus-image to output a new bootstrap along with an independent prefetchblob; 2. nydusify generate&push new meta layer including new bootstrap and the prefetch-files , also generates&push new manifest/config/prefetchblob, completing the incremental image build. Signed-off-by: Xing Ma <[email protected]>
Maxing1997
force-pushed
the
maxing_mr/nydusify_optimize
branch
from
08bff89
to
8437d8d
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1649 +/- ##
==========================================
- Coverage 60.10% 59.98% -0.13%
==========================================
Files 147 147
Lines 49513 49603 +90
Branches 46994 47057 +63
==========================================
- Hits 29762 29756 -6
- Misses 17970 18065 +95
- Partials 1781 1782 +1
|
imeoer
reviewed
Dec 4, 2024
| .help("Output path of the new bootstrap"), | ||
| ) | ||
| .arg( | ||
| Arg::new("output-path") |
There was a problem hiding this comment.
Can we name it output-json and reuse the BuildOutput struct in builder/src/core/context.rs ?
| Arg::new("output-path") | ||
| .long("output-path") | ||
| .short('o') | ||
| .help("Output path of for save hot blob id"), |
There was a problem hiding this comment.
File path to save optimization information in JSON format
| @@ -79,6 +79,8 @@ bitflags! { | |||
| const _V5_NO_EXT_BLOB_TABLE = 0x8000_0000; | |||
| /// Blob is generated with chunkdict. | |||
| const IS_CHUNKDICT_GENERATED = 0x0000_0200; | |||
| /// Blob is a hot blob | |||
| const IS_PREFETCHBLOB = 0x0001_0000; | |||
There was a problem hiding this comment.
How about IS_SEPARATED_WITH_PREFETCH_FILES ?
| @@ -79,6 +79,8 @@ bitflags! { | |||
| const _V5_NO_EXT_BLOB_TABLE = 0x8000_0000; | |||
| /// Blob is generated with chunkdict. | |||
| const IS_CHUNKDICT_GENERATED = 0x0000_0200; | |||
| /// Blob is a hot blob | |||
There was a problem hiding this comment.
How about Blob is generated with separated prefetch files..
| @@ -508,6 +510,12 @@ impl BlobInfo { | |||
| } | |||
| } | |||
|
|
|||
| pub fn set_hot_blob_feature(&mut self, is_prefetchblob: bool) { | |||
There was a problem hiding this comment.
set_separated_with_prefetch_files_feature
| @@ -0,0 +1,536 @@ | |||
| // Copyright 2023 Nydus Developers. All rights reserved. | |||
| return nil | ||
| } | ||
|
|
||
| // Optimize covert and push a new optimized nydus image |
| return errors.Wrap(err, "Init source image parser") | ||
| } | ||
| sourceParser, err := parser.New(sourceRemote, runtime.GOARCH) | ||
| if sourceParser == nil { |
| return &blobDesc, nil | ||
| } | ||
|
|
||
| func pushCompoundBootstrap(ctx context.Context, opt Opt, buildInfo BuildInfo) (*bootstrapInfo, error) { |
There was a problem hiding this comment.
pushCompoundBootstrap -> pushNewBootstrap ?
| bootstrapTarRa, err := os.Open(bootstrapTarPath) | ||
| if err != nil { | ||
| return nil, errors.Wrap(err, "open bootstrap tar file") | ||
| } |
There was a problem hiding this comment.
defer bootstrapTarRa.Close()
|
Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant Issue #1589
The objective is to generate a new image, which references a new blob included prefetch file chunks. This will accelerate an startup performance due to high-priority pulling of prefetch blob.
Details
Implement optimize subcommand of nydusify to generate a new image, which references a new blob included prefetch file chunks.
Types of changes
What types of changes does your PullRequest introduce? Put an
xin all the boxes that apply:Checklist
Go over all the following points, and put an
xin all the boxes that apply.