Support embedded overlay in --oci mode (from SIF)

[dtrudg]

In native mode, an overlay can be embedded in a SIF image, and automatically r/o overlayed onto the container rootfs.

--writable enters the container with the overlay in r/w mode.

Support this functionality in --oci mode.

Requires work to specify:

• How an overlay image is stored in an oci-sif
• How an overlay image is associated with a rootfs in an oci-sif

---

From conversation with @tri-adam and @wobito it appears that a sensible implementation would likely involve the following implementation / considerations:

• The singularity overlay command is extended so that it is able to add an extfs image inside of an OCI-SIF file that already holds an OCI image consisting of single / multiple squashfs layers. The overlay is conceptually an additional layer on the image.

• The image manifest / config will be updated, and will be correct at the point the overlay is initially added. However, as the overlay is modified the manifest / config will no longer be correct as the true content hash of the overlay will change as it is modified.

• We must have a way to reconcile the manifest / config with updated overlay content - this effectively makes a new OCI image within the OCI-SIF.

• We must perform the reconciliation, at minimum, when an image is pushed to a registry for distribution. It could be possible to update the manifest/config whenever a container execution that modifies the r/w overlay exits... but this is potentially problematic where jobs may be killed, OCI-SIF files are stored on cluster-wide filesystems etc.

• When an OCI-SIF containing an extfs overlay layer is run, the behaviour is the same as in native mode. Default r/o overlay. Writable with --writable.

• When an image containing an overlay is pushed to an OCI registry then by default we can keep the r/w extfs overlay as a layer. The image will not be able to be consumed by runtimes other than Singularity.

• We might want to extend the singularity overlay command so that an extfs overlay in an OCI-SIF can manually be converted to a read-only squashfs layer. This effectively 'seals' the overlay and finalises the image with the changes that have been made in the overlay.

• When an image containing an overlay is pushed to an OCI registry, and the user requests it to be pushed as a 'true' OCI image (i.e. with tar layers runnable with a standard OCI runtime)... the extfs overlay must become a tar layer. This also effectively 'seals' the overlay and finalises the image with the changes that have been made in the overlay. Do we want the sealing in this case to be automatic, or require the user to seal the image first?

---

Sub-tasks:

• add extfs overlay to OCI-SIF via singularity overlay #2867
• default r/o mount of extfs overlay in --oci mode #2868
• optional --writable mount of extfs overlay in --oci mode #2869
• singularity overlay seal conversion of overlay to r/o layer in OCI-SIF #2870
• prevent push with tar layers for SIF with overlay #2871
• Update manifest / config for correct overlay digest after changes #3119