Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize RAFS filesystem builder #1257

Merged
merged 2 commits into from
May 23, 2023
Merged

Optimize RAFS filesystem builder #1257

merged 2 commits into from
May 23, 2023

Conversation

jiangliu
Copy link
Collaborator

rafs: optimize the way to build RAFS filesystem

The current way to build RAFS filesystem is:

  • build the lower tree from parent bootstrap
  • convert the lower tree into an array
  • build the upper tree from source
  • merge the upper tree into the lower tree
  • convert the merged tree into another array
  • dump nodes from the array

Now we optimize it as:

  • build the lower tree from parent bootstrap
  • build the upper tree from source
  • merge the upper tree into the lower tree
  • dump the merged tree

@jiangliu jiangliu requested a review from a team as a code owner April 30, 2023 02:28
@jiangliu jiangliu requested review from imeoer, changweige and adamqqqplay and removed request for a team April 30, 2023 02:28
@anolis-bot
Copy link
Collaborator

@jiangliu , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/70499

@codecov
Copy link

codecov bot commented Apr 30, 2023 •
edited
Loading

Codecov Report

Merging #1257 (9c0f502) into master (17fd41c) will decrease coverage by 0.29%.
The diff coverage is 34.17%.

❗ Current head 9c0f502 differs from pull request most recent head fc346b4. Consider uploading reports for the commit fc346b4 to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1257      +/-   ##
==========================================
- Coverage   45.70%   45.41%   -0.29%     
==========================================
  Files         124      126       +2     
  Lines       36884    37465     +581     
  Branches    36884    37465     +581     
==========================================
+ Hits        16856    17016     +160     
- Misses      19139    19557     +418     
- Partials      889      892       +3
Impacted Files Coverage Δ
rafs/src/builder/compact.rs 0.00% <0.00%> (ø)
rafs/src/builder/core/layout.rs 0.00% <0.00%> (ø)
rafs/src/builder/core/overlay.rs 0.00% <ø> (-13.59%) ⬇️
rafs/src/builder/core/v6.rs 32.37% <0.00%> (+0.34%) ⬆️
rafs/src/builder/directory.rs 0.00% <0.00%> (ø)
rafs/src/builder/merge.rs 0.00% <0.00%> (ø)
rafs/src/builder/stargz.rs 0.00% <0.00%> (ø)
src/bin/nydus-image/main.rs 40.91% <0.00%> (ø)
src/bin/nydus-image/stat.rs 0.00% <0.00%> (ø)
src/bin/nydus-image/validator.rs 0.00% <0.00%> (ø)
... and 13 more

... and 10 files with indirect coverage changes

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/70500

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/70548

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/70561

@anolis-bot
Copy link
Collaborator

@yqleng1987 , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/73860

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/73886

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

imeoer
imeoer reviewed May 19, 2023
View reviewed changes
Copy link
Collaborator

@imeoer imeoer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the hard work! Do we have a perf comparison? For example, convert a wordpress image.

rafs/src/builder/core/context.rs Outdated Show resolved Hide resolved
rafs/src/builder/core/tree.rs Outdated Show resolved Hide resolved
@jiangliu
rafs: rename set_4k_aligned() to set_aligned()
41fdfa3 
Rename set_4k_aligned() to set_aligned(), for tarfs.

Signed-off-by: Jiang Liu <[email protected]>
@jiangliu
rafs: optimize the way to build RAFS filesystem
fc346b4 
The current way to build RAFS filesystem is:
- build the lower tree from parent bootstrap
- convert the lower tree into an array
- build the upper tree from source
- merge the upper tree into the lower tree
- convert the merged tree into another array
- dump nodes from the array

Now we optimize it as:
- build the lower tree from parent bootstrap
- build the upper tree from source
- merge the upper tree into the lower tree
- dump the merged tree

Signed-off-by: Jiang Liu <[email protected]>
@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/74676

@jiangliu
Copy link
Collaborator Author

Thanks for the hard work! Do we have a perf comparison? For example, convert a wordpress image.

There is another PR to improve the performance, will do test with the coming PR:)

@imeoer imeoer merged commit 809f8d9 into dragonflyoss:master May 23, 2023
@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@adamqqqplay adamqqqplay mentioned this pull request May 25, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bergwolf bergwolf bergwolf left review comments

@imeoer imeoer imeoer approved these changes

@changweige changweige Awaiting requested review from changweige changweige is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@adamqqqplay adamqqqplay Awaiting requested review from adamqqqplay adamqqqplay is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@liubin liubin Awaiting requested review from liubin liubin was automatically assigned from dragonflyoss/nydus-maintainers

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jiangliu @anolis-bot @yqleng1987 @bergwolf @imeoer