Optional conversion from AUFS to OverlayFS whiteout format in tar -> squashfs #25
Assignees
Labels
enhancement
New feature or request
Comments
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 18, 2023
Add an option, `OptConvertWhiteout`, for `squashfsConvertor`, that when set true will convert AUFS whiteout files in the incoming tar, to OverlayFS whiteouts in the outgoing squashfs. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 18, 2023
Add an option, `OptConvertWhiteout`, for `squashfsConverter`, that when set true will convert AUFS whiteout files in the incoming tar, to OverlayFS whiteouts in the outgoing squashfs. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 18, 2023
Add an option, `OptConvertWhiteout`, for `squashfsConverter`, that when set true will convert AUFS whiteout files in the incoming tar, to OverlayFS whiteouts in the outgoing squashfs. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 18, 2023
Add an option, `OptConvertWhiteout`, for `squashfsConverter`, that when set true will convert AUFS whiteout files in the incoming tar, to OverlayFS whiteouts in the outgoing squashfs. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 18, 2023
Add an option, `OptConvertWhiteout`, for `squashfsConverter`, that when set true will convert AUFS whiteout files in the incoming tar, to OverlayFS whiteouts in the outgoing squashfs. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 19, 2023
By default, replace AUFS whiteout markers with OverlayFS whiteout markers when converting a layer from TAR to SquashFS. It is expected that SquashFS layers will be used via direct mount, rather than extraction to disk. Therefore, the whiteout markers must be in a format that supports assembling multiple mounted layers into a rootfs, using OverlayFS. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. A new option `OptNoConvertWhiteout` will disable whiteout conversion. This can be employed by callers to avoid unneccessary processing where the layer is part of a squashed / single-layer image that will not have any whiteout markers. Fixes sylabs#25
dtrudg
added a commit
to dtrudg/oci-tools
that referenced
this issue
Oct 19, 2023
By default, replace AUFS whiteout markers with OverlayFS whiteout markers when converting a layer from TAR to SquashFS. It is expected that SquashFS layers will be used via direct mount, rather than extraction to disk. Therefore, the whiteout markers must be in a format that supports assembling multiple mounted layers into a rootfs, using OverlayFS. A `.wh.file` marker indicating a whiteout of `file` is replaced by `file` as a 0:0 character device. A `dir/.wh..wh..opq` marker indicating an opaque directory is replaced with the `trusted.overlay.opaque="y"` xattr on `dir`. A two-pass approach is required. AUFS directory opaque markers are not required to be adjacent to the directory to which they apply, in the tar stream. As `tar2sqfs` and `sqfstar` do not support tar files with two instances of the same dir/file, we cannot append an additional tar entry if we come across an opaque marker distant from the target directory entry. A new option `OptNoConvertWhiteout` will disable whiteout conversion. This can be employed by callers to avoid unneccessary processing where the layer is part of a squashed / single-layer image that will not have any whiteout markers. Fixes sylabs#25
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If a standard OCI image, with tar layers, is converted to squashfs layer format then it is likely that the resulting squashfs layers will be combined into a container root filesystem using OverlayFS. This will be the case for SingularityCE.
The image-spec tar layer format specifies AUFS style whiteout/opaque markers for files/directories that have been deleted between layers:
These markers are not understood by OverlayFS, which uses different markers:
There is no practical way to convert the markers from AUFS -> OverlayFS format at mount time. Therefore, oci-tools must be able to convert from AUFS -> OverlayFS format in the tar -> squashfs conversion.
The text was updated successfully, but these errors were encountered: