Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API(s) to support modification #48

Closed
tri-adam opened this issue May 3, 2024 · 2 comments · Fixed by #62
Closed

API(s) to support modification #48

tri-adam opened this issue May 3, 2024 · 2 comments · Fixed by #62
Labels
enhancement New feature or request

Comments

@tri-adam
Copy link
Member

tri-adam commented May 3, 2024

Spec out one or more API(s) to support the addition of an overlay, and to update the relevant image manifest/config and root index after an overlay is modified (sylabs/singularity#1480). Also see discussion in #46.
@tri-adam tri-adam added the enhancement New feature or request label May 3, 2024
@dtrudg
Copy link
Member

dtrudg commented Jun 17, 2024 •
edited
Loading

Assuming we are following the flow for Singularity detailed in sylabs/singularity#1480 we need to be able to perform 2 modification operations for minimal functionality, with an additional 'seal' operation if we wish to allow images with overlays to be finalized and / or pushed to an OCI registry as an OCI image with r/o tar layers.

1. Add EXTFS Overlay

We must be able to add an extfs overlay, of configurable size, to the SIF file. The overlay will be added to the image config as an additional layer with an extfs mediaType. The image config, manifest, and RootIndex will be updated so that all hashes are consistent at the time of insertion of the extfs overlay layer.

To avoid re-writing the entirity of the SIF file, we would need to enforce that all existing image layers are located at the start of the SIF file. Large images can have many and large layers, so we don't want to re-write the whole SIF if at all possible.

For an image with N layers - Before:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob Image Config
N+2 OCI.Blob Image Manifest
N+3 OCI.RootIndex Image Index

After the addition of an extfs overlay:

descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob EXTFS overlay layer New
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

2. Update Metadata

As the EXTFS overlay layer is modified, the image config, manifest, and RootIndex will become inconsistent w.r.t. the hash for the extfs overlay layer. Due to concurrency issues, the risk of batch jobs being killed etc. in the SingularityCE use case it is probably not practical to update the image config, manifest, root index after each use of the overlay in r/w mode. However, before we share the image in a registry, we need the hashes to be correct.

We must have an operation that updates the image metadata, rewriting image config, manifest, rootindex appropriately.

Before metadata is updated:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob EXTFS overlay layer
N+2 OCI.Blob Image Config
N+3 OCI.Blob Image Manifest
N+4 OCI.RootIndex Image Index

After metadata is updated:

descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob EXTFS overlay layer Unmodified
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

3. 'Seal' Overlay

When a user has finished modifying an image via an overlay, we may wish to allow them be able to 'seal' the image. This will convert the EXTFS overlay layer into a SquashFS layer.

Sealing an overlay - into a r/o layer - is also a prerequisite for being able to subsequently push the image in tar layer format, compatibile with other runtimes.

Before image is sealed:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob EXTFS overlay layer
N+2 OCI.Blob Image Config
N+3 OCI.Blob Image Manifest
N+4 OCI.RootIndex Image Index
descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob SquashFS layer New - conversion of extfs layer
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

@tri-adam
Copy link
Member Author

1. Add EXTFS Overlay

We must be able to add an extfs overlay, of configurable size, to the SIF file. The overlay will be added to the image config as an additional layer with an extfs mediaType. The image config, manifest, and RootIndex will be updated so that all hashes are consistent at the time of insertion of the extfs overlay layer.

To avoid re-writing the entirity of the SIF file, we would need to enforce that all existing image layers are located at the start of the SIF file. Large images can have many and large layers, so we don't want to re-write the whole SIF if at all possible.

For an image with N layers - Before:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob Image Config
N+2 OCI.Blob Image Manifest
N+3 OCI.RootIndex Image Index
After the addition of an extfs overlay:

descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob EXTFS overlay layer New
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

I think there are a couple of steps here. At a minimum:

  • Construct an updated v1.Image with the EXTFS layer and updated image config/manifest. mutate.AppendLayers can be used for this purpose. You would need to construct a v1.Layer containing the EXTFS filesystem, which is probably best done in the SingularityCE code base?
  • Replace the original v1.Image with the updated v1.Image. Here, we probably need an API in github.com/sylabs/oci-tools/pkg/sif. Perhaps something like:
func Update(path string, ii v1.ImageIndex, opts ...UpdateOpt) error

Update would expect an OCI-SIF at path and replace its rootIndex with ii. The logic for this could be something like:

  • Generate a list of all OCIBlob digests referenced by ii, and digest of new rootIndex.
  • Remove all SIF descriptors not in the list of references (including rootIndex if it's been updated.)
  • For each descriptor in the list of references, check if the blob is already in the SIF. If not, add it to the SIF.

2. Update Metadata

As the EXTFS overlay layer is modified, the image config, manifest, and RootIndex will become inconsistent w.r.t. the hash for the extfs overlay layer. Due to concurrency issues, the risk of batch jobs being killed etc. in the SingularityCE use case it is probably not practical to update the image config, manifest, root index after each use of the overlay in r/w mode. However, before we share the image in a registry, we need the hashes to be correct.

We must have an operation that updates the image metadata, rewriting image config, manifest, rootindex appropriately.

Before metadata is updated:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob EXTFS overlay layer
N+2 OCI.Blob Image Config
N+3 OCI.Blob Image Manifest
N+4 OCI.RootIndex Image Index
After metadata is updated:

descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob EXTFS overlay layer Unmodified
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

This should be fairly do-able. Again, you'll need to construct an updated v1.Image, this time that contains an updated EXTFS v1.Layer. This should just be a matter of calculating the layer digest, constructing a v1.Layer similarly to the previous step, and then using mutate.SetLayer to replace the old layer.

From there, if you call Update as described above, I think it might just work?

3. 'Seal' Overlay

When a user has finished modifying an image via an overlay, we may wish to allow them be able to 'seal' the image. This will convert the EXTFS overlay layer into a SquashFS layer.

Sealing an overlay - into a r/o layer - is also a prerequisite for being able to subsequently push the image in tar layer format, compatibile with other runtimes.

Before image is sealed:

descriptor id type content
1 OCI.Blob SquashFS layer
... OCI.Blob SquashFS layer
N OCI.Blob SquashFS layer
N+1 OCI.Blob EXTFS overlay layer
N+2 OCI.Blob Image Config
N+3 OCI.Blob Image Manifest
N+4 OCI.RootIndex Image Index
descriptor id type content state
1 OCI.Blob SquashFS layer Unmodified
... OCI.Blob SquashFS layer Unmodified
N OCI.Blob SquashFS layer Unmodified
N+1 OCI.Blob SquashFS layer New - conversion of extfs layer
N+2 OCI.Blob Image Config Updated
N+3 OCI.Blob Image Manifest Updated
N+4 OCI.RootIndex Image Index Updated

This will again follow a similar flow I think. The tricky part here will be the EXTFS->SquashFS conversion, and I'm not sure if this belongs in this repo or in SingularityCE. I'd lean towards CE for now, as EXTFS seems fairly specific to that at the moment?

Once you have a v1.Layer containing the result of the conversion... it should be a matter of mutate.SetLayer and then Update again?

@tri-adam tri-adam mentioned this issue Jun 18, 2024
Merged
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jun 25, 2024
@dtrudg
feat: Update SIF from ImageIndex
de42d45 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jun 25, 2024
@dtrudg
feat: Update SIF from ImageIndex
fbf2ea4 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
@dtrudg dtrudg mentioned this issue Jun 25, 2024
Merged
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jul 4, 2024
@dtrudg
feat: Update SIF from ImageIndex
68f53ad 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jul 4, 2024
@dtrudg
feat: Update SIF from ImageIndex
1ac46a9 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jul 8, 2024
@dtrudg
feat: Update SIF from ImageIndex
8e2a254 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jul 8, 2024
@dtrudg
feat: Update SIF from ImageIndex
e6b37ce 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
dtrudg added a commit to dtrudg/oci-tools that referenced this issue Jul 8, 2024
@dtrudg
feat: Update SIF from ImageIndex
d57ab7f 
Add `sif.Update` which accepts updates a SIF so that its content
reflects the ImageIndex passed.

Any blobs in the SIF that are not present in the new ImageIndex will be
removed from the SIF.

Any blobs that are present in the new ImageIndex, but not in the SIF,
will be added to the SIF.

Blobs that are present in both the SIF and the new ImageIndex are not
re-written. They will remain at their current descriptor location in the
SIF.

Closes sylabs#48
@dtrudg dtrudg closed this as completed in #62 Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Update SIF from ImageIndex dtrudg/oci-tools
2 participants
@dtrudg @tri-adam