Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add verity digests for exported block device #1176

Merged
merged 2 commits into from
Mar 31, 2023
Merged

Add verity digests for exported block device #1176

merged 2 commits into from
Mar 31, 2023

Conversation

jiangliu
Copy link
Collaborator

@jiangliu jiangliu commented Mar 27, 2023 •
edited
Loading

Add --verity option to nydus-image export --block to generate
dm-verity data for block devices.

[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# tar -cvf src.tar src
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# sha256sum src.tar
0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a  src.tar
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# cp src.tar images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# target/debug/nydus-image create -t tar-tarfs -D images/ images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
[2023-03-27 16:32:00.068730 +08:00] INFO successfully built RAFS filesystem:
meta blob path: images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
data blob size: 0x3c000
data blobs: ["0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a"]
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# target/debug/nydus-image export --block --verity -D images/ -B images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47 
[2023-03-28 11:16:31.953692 +08:00] INFO RAFS features: COMPRESSION_NONE | HASH_SHA256 | EXPLICIT_UID_GID | TARTFS_MODE
dm-verity options: --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 4576 --hash-offset 2342912 6b5743e7da406a33ab3a8bb03b65e67d1c1951b2d7ebc5026e0de3fb44a7cc20
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# losetup -r /dev/loop1 images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# veritysetup open --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 4576 --hash-offset 2342912 /dev/loop1 verity /dev/loop1 6b5743e7da406a33ab3a8bb03b65e67d1c1951b2d7ebc5026e0de3fb44a7cc20
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# veritysetup status verity
/dev/mapper/verity is active and is in use.
  type:        VERITY
  status:      verified
  hash type:   1
  data block:  512
  hash block:  4096
  hash name:   sha256
  salt:        -
  data device: /dev/loop1
  data loop:   /root/image-service/images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
  size:        4576 sectors
  mode:        readonly
  hash device: /dev/loop1
  hash loop:   /root/image-service/images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
  hash offset: 4576 sectors
  root hash:   6b5743e7da406a33ab3a8bb03b65e67d1c1951b2d7ebc5026e0de3fb44a7cc20
mount -t erofs /dev/dm-0 mnt

@jiangliu jiangliu requested a review from a team as a code owner March 27, 2023 16:01
@jiangliu jiangliu requested review from bergwolf, luodw and adamqqqplay and removed request for a team March 27, 2023 16:01
@anolis-bot
Copy link
Collaborator

@jiangliu , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/61944

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/61945

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
nydus_ci❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/61969

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/61979

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@anolis-bot
Copy link
Collaborator

@jiangliu , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/61982

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@jiangliu
utils: introduce mechanism to generate Merkle tree for verity
c6d2065 
Introduce mechanism to generate Merkle tree for verity.

Signed-off-by: Jiang Liu <[email protected]>
@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/62763

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/62866

@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/62870

@jiangliu
nydus-image: generate dm-verity data for block device
01e59a6 
Add `--verity` option to `nydus-image export --block` to generate
dm-verity data for block devices.

```
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# tar -cvf src.tar src
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# sha256sum src.tar
0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a  src.tar
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# cp src.tar images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# target/debug/nydus-image create -t tar-tarfs -D images/ images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
[2023-03-27 16:32:00.068730 +08:00] INFO successfully built RAFS filesystem:
meta blob path: images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
data blob size: 0x3c000
data blobs: ["0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a"]
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# target/debug/nydus-image export --block --verity -D images/ -B images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
[2023-03-27 23:49:14.450762 +08:00] INFO RAFS features: COMPRESSION_NONE | HASH_SHA256 | EXPLICIT_UID_GID | TARTFS_MODE
dm-verity options: --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=4096 --hash-block-size=4096 --data-blocks 572 --hash-offset 2342912 ab7b417fc284c3b58a72044a996ec55e2c68a8b9dcf10bc469f4e640e5d98e6a
losetup -r /dev/loop1 images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# veritysetup open -v --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=4096 --hash-block-size=4096 --data-blocks 572 --hash-offset 2342912 /dev/loop1 verity /dev/loop1 ab7b417fc284c3b58a72044a996ec55e2c68a8b9dcf10bc469f4e640e5d98e6a
[root@iZ0jl3vazmhc81dur3xnm3Z image-service]# veritysetup status verity
/dev/mapper/verity is active.
  type:        VERITY
  status:      verified
  hash type:   1
  data block:  4096
  hash block:  4096
  hash name:   sha256
  salt:        -
  data device: /dev/loop1
  data loop:   /root/image-service/images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
  size:        4576 sectors
  mode:        readonly
  hash device: /dev/loop1
  hash loop:   /root/image-service/images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
  hash offset: 4576 sectors
  root hash:   ab7b417fc284c3b58a72044a996ec55e2c68a8b9dcf10bc469f4e640e5d98e6a
```

Signed-off-by: Jiang Liu <[email protected]>
@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/62874

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

liubogithub
liubogithub approved these changes Mar 31, 2023
View reviewed changes
Copy link
Collaborator

@liubogithub liubogithub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

otherwise looks good.

service/src/block_device.rs
&buf[offset..offset + block_size],
digest::Algorithm::Sha256,
);
let mut guard = generator.lock().unwrap();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to put generator.lock.unwrap() out of the while loop?

@jiangliu jiangliu merged commit b79e90b into dragonflyoss:master Mar 31, 2023
@imeoer
Copy link
Collaborator

imeoer commented Mar 31, 2023

Can we add the usage in the docs?

@hangvane hangvane mentioned this pull request Apr 14, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liubogithub liubogithub liubogithub approved these changes

@bergwolf bergwolf Awaiting requested review from bergwolf bergwolf is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@luodw luodw Awaiting requested review from luodw luodw is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@adamqqqplay adamqqqplay Awaiting requested review from adamqqqplay adamqqqplay is a code owner automatically assigned from dragonflyoss/nydus-reviewers

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jiangliu @anolis-bot @imeoer @liubogithub