images: base image specific toybox config and version updates

[mythi]

Fixes: #1506

Our *.Dockefile's allow builders to specify FINAL_BASE base image to which static binaries are copied for the final image.

Initcontainers install toybox with a minimum set of "coreutils" needed but with some final base images the minimum set can be even smaller since they may already have /bin/sh, ls etc. installed.

Make initcontainers to have base image specific toybox-config. For now, "static" for distroles/static and "ubi8-micro" for UBI base image configs are created.

At the same time, move the toybox build to "usrmerge" based setup where everything is installed to /usr/bin and /bin is a symlink to /usr/bin.

[codecov-commenter]

Codecov Report

Merging #1525 (ec96757) into main (dfe75b1) will not change coverage.
The diff coverage is n/a.

❗ Current head ec96757 differs from pull request most recent head ad6c524. Consider uploading reports for the commit ad6c524 to get more accurate results

@@           Coverage Diff           @@
##             main    #1525   +/-   ##
=======================================
  Coverage   50.04%   50.04%           
=======================================
  Files          43       43           
  Lines        4884     4884           
=======================================
  Hits         2444     2444           
  Misses       2301     2301           
  Partials      139      139           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

📢 Have feedback on the report? Share it here.

[mythi]

@mregmi can you please check this on your side using the build flow you have

[mregmi]

@chaitanya1731 can you test this for OCP. Thanks

[chaitanya1731]

Hi @mythi, we tried to test this functionality with sgx initcontainer using make intel-sgx-initcontainer and found some errors - We got multiple warnings about SHELL is not supported for OCI image format, [/bin/bash -o pipefail -c] will be ignored. Must use 'docker' format and other linking error with the message - collect2: error: ld returned 1 exit status.
See logs below -

[1/2] STEP 18/20: SHELL ["/bin/bash", "-o", "pipefail", "-c"]
WARN[0055] SHELL is not supported for OCI image format, [/bin/bash -o pipefail -c] will be ignored. Must use `docker` format 
--> 61d8af5d6f9
[1/2] STEP 19/20: ARG FINAL_BASE=registry.access.redhat.com/ubi9-micro:latest
WARN[0055] SHELL is not supported for OCI image format, [/bin/bash -o pipefail -c] will be ignored. Must use `docker` format 
--> 7a1aa17434c
[1/2] STEP 20/20: RUN curl -SL https://github.com/landley/toybox/archive/refs/tags/$TOYBOX_VERSION.tar.gz -o toybox.tar.gz     && echo "$TOYBOX_SHA256 toybox.tar.gz" | sha256sum -c -     && tar -xzf toybox.tar.gz     && rm toybox.tar.gz     && cd toybox-$TOYBOX_VERSION     && KCONFIG_CONFIG=${DIR}/build/docker/toybox-config-$(echo ${FINAL_BASE} | xargs basename -s :latest) LDFLAGS="--static" CC=musl-gcc PREFIX=$ROOT/usr/bin V=2 make toybox install_flat     && cd $ROOT && ln -fs usr/bin bin && cd -     && install -D LICENSE $ROOT/licenses/toybox     && cp -r /usr/share/doc/musl $ROOT/licenses/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3459k    0 3459k    0     0  4838k      0 --:--:-- --:--:-- --:--:-- 4838k
toybox.tar.gz: OK
scripts/genconfig.sh
scripts/make.sh

Which C files to build...
Library probemusl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lutil -o generated/unstripped/libprobe
echo -n  -lutil
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lcrypt -o generated/unstripped/libprobe
echo -n  -lcrypt
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lm -o generated/unstripped/libprobe
echo -n  -lm
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lresolv -o generated/unstripped/libprobe
echo -n  -lresolv
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lselinux -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lselinux
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lsmack -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lsmack
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lattr -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lattr
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lcrypto -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lcrypto
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lz -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -llog -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -llog
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -liconv -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -liconv
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -ltls -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -ltls
collect2: error: ld returned 1 exit status
musl-gcc -Wall -Wundef -Werror=implicit-function-declaration -Wno-char-subscripts -Wno-pointer-sign -funsigned-char --static -xc - -lssl -o generated/unstripped/libprobe
/usr/bin/ld: cannot find -lssl
collect2: error: ld returned 1 exit status

generated/{Config.in,newtoys.hcc scripts/mkflags.c -o generated/unstripped/mkflags
,flags.h,globals.hcc scripts/mktags.c -o generated/unstripped/mktags
,tags.hcc scripts/config2help.c -o generated/unstripped/config2help
,help.h}
Compile toybox

However the image is still built and upon doing RH container preflight certification tests, we got same error like before about modification of the base UBI image -

time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=bin layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file="usr/bin/[" layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/tee layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/sh layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/test layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/cp layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/head layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/echo layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/ln layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/basename layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/realpath layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/bash layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/cut layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/ls layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="found disallowed modification in layer" file=usr/bin/cat layer="sha256:a06459a9df6cbdef37161ec72bca50de6837b8a84b8f4b08c6ebd199c853e827"
time="2023-09-07T12:58:06-04:00" level=info msg="check completed" check=HasModifiedFiles result=FAILED

 "failed": [
            {
                "name": "HasModifiedFiles",
                "elapsed_time": 1458,
                "description": "Checks that no files installed via RPM in the base Red Hat layer have been modified",
                "help": "Check HasModifiedFiles encountered an error. Please review the preflight.log file for more information.",
                "suggestion": "Do not modify any files installed by RPM in the base Red Hat layer",
                "knowledgebase_url": "https://access.redhat.com/documentation/en-us/red_hat_software_certification/8.63/html-single/red_hat_openshift_software_certification_policy_guide/index#assembly-requirements-for-container-images_openshift-sw-cert-policy-introduction",
                "check_url": "https://access.redhat.com/documentation/en-us/red_hat_software_certification/8.63/html-single/red_hat_openshift_software_certification_policy_guide/index#assembly-requirements-for-container-images_openshift-sw-cert-policy-introduction"
            }

Could you please take a look. Thanks.

[chaitanya1731]

found this thread about the WARN for /bin/bash. looks like it is not supported by podman build yet.. containers/podman#8477

[mythi]

@chaitanya1731 the collect2 errors are not relevant. I also think that SHELL thing is just a warning. What's strange is that it seems the uses "toybox-config-static" file because you get all the symlinks. For toybox-config-ubi9-micro it should only give grep and lspci.

[mythi]

using make intel-sgx-initcontainer and found some errors

So this is the problem. make intel-sgx-initcontainer does not give you the UBI based final image but it's distroless and that's why you get the certification errors.

[mythi]

@chaitanya1731 while we merged this, I'd still be happy to get your report how this works with ubi9-micro and wrt. the certification. I will do additional fixing if needed.

[chaitanya1731]

So this is the problem. make intel-sgx-initcontainer does not give you the UBI based final image but it's distroless and that's why you get the certification errors.

@mythi about this, we tested with the ubi9-micro image.. Just commented the line 32 in build-image.sh where it sets the final base image ARG using --build-arg to distroless. Also in the logs attached above, if you see 3rd line it pulls the ubi9-micro

[1/2] STEP 19/20: ARG FINAL_BASE=registry.access.redhat.com/ubi9-micro:latest

[mythi]

I see. I tested this with ubi8 but I need to double check. Can you check the right toybox config gets used?

[mythi]

So this is the problem. make intel-sgx-initcontainer does not give you the UBI based final image but it's distroless and that's why you get the certification errors.

@mythi about this, we tested with the ubi9-micro image.. Just commented the line 32 in build-image.sh where it sets the final base image ARG using --build-arg to distroless. Also in the logs attached above, if you see 3rd line it pulls the ubi9-micro

[1/2] STEP 19/20: ARG FINAL_BASE=registry.access.redhat.com/ubi9-micro:latest

The expected flow works OK for me:

$ DOCKER_BUILDKIT=0  docker build -t ubi-test:devel -f build/docker/intel-qat-initcontainer.Dockerfile .
$ docker run -it --entrypoint /bin/bash ubi-test:devel
bash-5.1# realpath /bin/sh
/usr/bin/bash
bash-5.1# realpath /usr/bin/ls
/usr/bin/ls
bash-5.1# 

[mythi]

The expected flow works OK for me:

@chaitanya1731 Would you be able to confirm? I also tested with podman on my side and I don't see anything wrong:

With:

diff --git a/build/docker/build-image.sh b/build/docker/build-image.sh
index f070a7b1..04466d45 100755
--- a/build/docker/build-image.sh
+++ b/build/docker/build-image.sh
@@ -29,7 +29,7 @@ if [ -d $(dirname $0)/../../vendor ] ; then
     BUILD_ARGS="${BUILD_ARGS} --build-arg DIR=/go/src/github.com/intel/intel-device-plugins-for-kubernetes --build-arg GO111MODULE=off"
 fi
 
-BUILD_ARGS="${BUILD_ARGS} --build-arg FINAL_BASE=gcr.io/distroless/static"
+BUILD_ARGS="${BUILD_ARGS}"
 if [ -z "${BUILDER}" -o "${BUILDER}" = 'docker' -o "${BUILDER}" = 'podman' ] ; then
     DOCKER_BUILDKIT=0 ${BUILDER} build --pull -t ${IMG}:${TAG} ${BUILD_ARGS} -f ${DOCKERFILE} .
 elif [ "${BUILDER}" = 'buildah' ] ; then

I can: make intel-qat-initcontainer BUILDER=podman and

$ podman run -it --entrypoint /bin/bash localhost/intel/intel-qat-initcontainer:devel
bash-5.1# realpath /usr/bin/ls
/usr/bin/ls

which suggest at least ls is not modified.

[chaitanya1731]

Hi @mythi , me and @mregmi double checked this on our ends and we still get the preflight issue.. the test fails with

time="2023-09-29T14:49:06-04:00" level=info msg="found disallowed modification in layer" file=bin layer="sha256:1b832d6082dc9059e99fdcf5945be9c472424a9688f59f16d1899d366c8229b2"
time="2023-09-29T14:49:06-04:00" level=info msg="check completed" check=HasModifiedFiles result=FAILED

Looks like its just one file thats being modified now bin, cant see the other errors with /usr/bin/ I am checking any other way to debug this further..

[mythi]

We re-create bin -> usr/bin symlink. Can you try ln without force (-f)?