OCI layout extensions #2633

vrothberg · 2024-11-14T12:07:37Z

Taken over from #2567 but:

Added tests for the three commits
Use reflinks when adding a file

vrothberg · 2024-11-14T12:07:46Z

In draft for now.

oci/layout/oci_dest.go

mtrmac

Just a quick skim; I’m afraid I didn’t read the added tests yet.

oci/layout/oci_dest.go

vrothberg · 2024-11-19T09:43:39Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works. We're using the function from c/storage which is e2e tested. The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

mtrmac

Actual detailed review still pending, I’m afraid.

mtrmac · 2024-12-02T22:59:51Z

oci/layout/oci_dest.go

@@ -18,10 +18,12 @@ import (
 	"github.com/containers/image/v5/internal/private"
 	"github.com/containers/image/v5/internal/putblobdigest"
 	"github.com/containers/image/v5/types"
+	reflinkCopy "github.com/containers/storage/drivers/copy"


This drags in quite a few dependencies:

+github.com/containers/storage/drivers/copy +github.com/containers/storage/pkg/archive +github.com/containers/storage/pkg/chrootarchive +github.com/containers/storage/pkg/idtools +github.com/containers/storage/pkg/ioutils +github.com/containers/storage/pkg/longpath +github.com/containers/storage/pkg/mount +github.com/containers/storage/pkg/pools +github.com/containers/storage/pkg/promise +github.com/containers/storage/pkg/reexec +github.com/containers/storage/pkg/system +github.com/containers/storage/pkg/unshare +# github.com/docker/go-units v0.5.0 +## explicit +github.com/docker/go-units +# github.com/json-iterator/go v1.1.12 +## explicit; go 1.12 +github.com/json-iterator/go +# github.com/klauspost/compress v1.17.11 +## explicit; go 1.21 +github.com/klauspost/compress +github.com/klauspost/compress/flate +github.com/klauspost/compress/fse +github.com/klauspost/compress/huff0 +github.com/klauspost/compress/internal/cpuinfo +github.com/klauspost/compress/internal/snapref +github.com/klauspost/compress/zstd +github.com/klauspost/compress/zstd/internal/xxhash +# github.com/klauspost/pgzip v1.2.6 +## explicit +github.com/klauspost/pgzip +# github.com/moby/sys/capability v0.3.0 +## explicit; go 1.21 +github.com/moby/sys/capability +# github.com/moby/sys/mountinfo v0.7.2 +## explicit; go 1.17 +github.com/moby/sys/mountinfo +# github.com/moby/sys/user v0.3.0 +## explicit; go 1.17 +github.com/moby/sys/user +# github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd +## explicit +github.com/modern-go/concurrent +# github.com/modern-go/reflect2 v1.0.2 +## explicit; go 1.12 +github.com/modern-go/reflect2 +# github.com/opencontainers/runtime-spec v1.2.0 +## explicit +github.com/opencontainers/runtime-spec/specs-go +# github.com/ulikunitz/xz v0.5.12 +## explicit; go 1.12 +github.com/ulikunitz/xz +github.com/ulikunitz/xz/internal/hash +github.com/ulikunitz/xz/internal/xlog +github.com/ulikunitz/xz/lzma

Podman itself does not care, but podman-remote might, and out-of-Podman OCI-specific utilities probably do.
Can only the relevant code be split into a subpackage in c/storage?

mtrmac · 2024-12-02T23:03:44Z

oci/layout/oci_dest.go

+}
+
+// PutBlobFromLocalFileOptions is unused but may receive functionality in the future.
+type PutBlobFromLocalFileOptions struct{}


Good idea, but please use functional options so that presence/absence, and error reporting, is explicit.

Modeling after signature/sigstore, the minimum might be

type PutBlobFromLocalFileOption struct{} // to later become an alias to a private, maybe function, type func PutBlobFromLocalFile(…, opts ... PutBlobFromLocalFileOption)

mtrmac

A full review now.

mtrmac · 2024-12-03T16:16:32Z

docs/containers-transports.5.md

@@ -71,13 +71,15 @@ An image stored in the docker daemon's internal storage.
 The image must be specified as a _docker-reference_ or in an alternative _algo_`:`_digest_ format when being used as an image source.
 The _algo_`:`_digest_ refers to the image ID reported by docker-inspect(1).

-### **oci:**_path_[`:`_reference_]
+### **oci:**_path_[`:`_reference_|@source-index}]_


Suggested change

### **oci:**_path_[`:`_reference_|@source-index}]_

### **oci:**_path_[`:`{_reference_|`@`_source-index_}]

mtrmac · 2024-12-03T16:21:34Z

oci/layout/oci_dest_test.go

@@ -178,3 +178,41 @@ func putTestManifest(t *testing.T, ociRef ociReference, tmpDir string) {
 	digest := digest.FromBytes(data).Encoded()
 	assert.Contains(t, paths, filepath.Join(tmpDir, "blobs", "sha256", digest), "The OCI directory does not contain the new manifest data")
 }
+
+func TestPutbloFromLocalFile(t *testing.T) {


Suggested change

func TestPutbloFromLocalFile(t *testing.T) {

func TestPutblobFromLocalFile(t *testing.T) {

mtrmac · 2024-12-03T16:28:17Z

oci/layout/oci_transport.go

 //
+// If sourceIndex==-1, the index will not be valid to point out the source image, only image will be used.
+// NewReference returns an OCI reference for a directory and a image.


This line no longer belongs here and can be deleted.

mtrmac · 2024-12-03T16:34:28Z

oci/layout/oci_transport_test.go

@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
 		"relativepath",


Holding the line at “all of *_transport.go should have test coverage”, please port the two test cases added to TestGetManifestDescriptor from #1677 (and then see if more need to be added).

mtrmac · 2024-12-03T16:35:58Z

oci/layout/oci_transport_test.go

@@ -253,12 +329,12 @@ func TestReferenceStringWithinTransport(t *testing.T) {
 }


TestReferenceStringWithinTransport should have a @sourceIndex test case (one exists in #1677 ).

mtrmac · 2024-12-03T17:08:22Z

oci/layout/reader.go

+// This file is named reader.go for consistency with other transports
+// handling of “image containers”, but we don’t  actually need a stateful reader object.


Suggested change

// This file is named reader.go for consistency with other transports

// handling of “image containers”, but we don’t actually need a stateful reader object.

// This file is named reader.go for consistency with other transports’

// handling of “image containers”, but we don’t actually need a stateful reader object.

mtrmac · 2024-12-03T17:09:32Z

oci/layout/reader_test.go

+
+	results, err := List("fixtures/i_do_not_exist")
+	require.Error(t, err)
+	require.Nil(t, results)


This value is conceptually “undefined” so I don’t think we need to test for it, but it also doesn’t really hurt.

mtrmac · 2024-12-03T17:11:54Z

oci/layout/reader_test.go

+			require.Equal(t, test.digests[i], res.ManifestDescriptor.Digest.String())
+			require.Equal(t, test.names[i], ociRef.image)
+			if test.names[i] != "" {
+				require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))


Suggested change

require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))

require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), ":"+ test.names[i]))

mtrmac · 2024-12-03T17:12:48Z

oci/layout/reader_test.go

+			require.Equal(t, test.names[i], ociRef.image)
+			if test.names[i] != "" {
+				require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))
+			}


else test that res.Reference is a sourceIndex reference with the right index?

Or, alternatively, use the returned reference to read a manifest and verify that it matches res.ManifestDescriptor.Digest.

mtrmac · 2024-12-03T17:14:41Z

oci/layout/oci_dest.go

+	}
+	blobName := blobFile.Name()
+
+	copyRange := false


Is there a reason we would not want to use this?

mtrmac · 2024-12-03T17:22:48Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works.

…

The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

Tests writing to non-test directories , and possibly leaving them around on aborts/failures, seems rather unexpected to me, some might consider it (or, at least, be worried that it is) hostile. So I’d prefer to leave this to e2e tests, maybe the future tests of the Podman artifact CLI.

(Just because I was looking it up, Google says reflinks don’t work on ext4. That’s not really a blocker, we could gate this behind a filesystem type check.)

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg mentioned this pull request Nov 14, 2024

WIP: oci/layout API extensions #2567

Closed

vrothberg commented Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

mtrmac reviewed Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

vrothberg force-pushed the layout-list branch 3 times, most recently from ae7457e to 234b45f Compare November 19, 2024 09:41

vrothberg marked this pull request as ready for review November 19, 2024 12:11

mtrmac added 3 commits November 19, 2024 13:11

Add @sourceIndex syntax to oci/layout

2d2d47b

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Add oci/layout.List

5944589

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Add oci/layout.PutBlobFromLocalFile

6481647

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg force-pushed the layout-list branch from 234b45f to 6481647 Compare November 19, 2024 12:12

mtrmac reviewed Dec 2, 2024

View reviewed changes

mtrmac reviewed Dec 3, 2024

View reviewed changes

vrothberg mentioned this pull request Dec 10, 2024

vendor c/image PR and update code baude/podman#40

Open

vrothberg added a commit to vrothberg/libpod that referenced this pull request Dec 10, 2024

vendor c/image PR and update code

ef6acd3

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCI layout extensions #2633

OCI layout extensions #2633

vrothberg commented Nov 14, 2024 •

edited

Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

mtrmac Dec 2, 2024

mtrmac Dec 2, 2024

mtrmac left a comment

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac Dec 3, 2024

mtrmac commented Dec 3, 2024

	### oci:_path_[`:`_reference_\|@source-index}]_
	### oci:_path_[`:`{_reference_\|`@`_source-index_}]

	func TestPutbloFromLocalFile(t *testing.T) {
	func TestPutblobFromLocalFile(t *testing.T) {

		@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
		"relativepath",

		@@ -253,12 +329,12 @@ func TestReferenceStringWithinTransport(t *testing.T) {
		}

		// This file is named reader.go for consistency with other transports
		// handling of “image containers”, but we don’t actually need a stateful reader object.

	require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))
	require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), ":"+ test.names[i]))

OCI layout extensions #2633

Are you sure you want to change the base?

OCI layout extensions #2633

Conversation

vrothberg commented Nov 14, 2024 • edited Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

Choose a reason for hiding this comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Dec 3, 2024

vrothberg commented Nov 14, 2024 •

edited

Loading