Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build multiarch bootstrap image #1149

Merged
merged 1 commit into from
Jul 27, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,15 @@ postsubmits:
memory: "1Gi"
limits:
memory: "1Gi"
- name: publish-bootstrap-image
- name: publish-multiarch-bootstrap-image
always_run: false
run_if_changed: "images/golang/.*|images/bootstrap/.*"
annotations:
testgrid-create-test-group: "false"
decorate: true
decoration_config:
grace_period: 5m0s
timeout: 4h0m0s
max_concurrency: 1
labels:
preset-dind-enabled: "true"
Expand All @@ -104,16 +107,16 @@ postsubmits:
- |
cat "$QUAY_PASSWORD" | docker login --username $(cat "$QUAY_USER") --password-stdin=true quay.io
cd images
./publish_image.sh bootstrap quay.io kubevirtci
./publish_image.sh golang quay.io kubevirtci
./publish_multiarch_image.sh bootstrap quay.io kubevirtci
./publish_multiarch_image.sh -l golang quay.io kubevirtci
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need this option?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point - we probably don't need this specific local image flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this flag. For Dockerfile whose baseimage is from remote repository, we need manually pull correct CPU arch base image, otherwise it would use local image even if the local image has different CPU arch with --platform.
For these Dockerfile we use local built image, like the golang image, we do not need to pull base image.

# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
memory: "1Gi"
memory: "8Gi"
limits:
memory: "1Gi"
memory: "8Gi"
- name: publish-kubekins-e2e-image
always_run: false
run_if_changed: "images/kubekins-e2e/.*"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,13 @@ presubmits:
memory: "1Gi"
limits:
memory: "1Gi"
- name: build-bootstrap-image
- name: build-multiarch-bootstrap-image
always_run: false
run_if_changed: "images/bootstrap/.*|images/golang/.*"
decorate: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we are doubling the number of images that we are building - we may need to update the default 2 hours timeout for this job. I tried running this locally and it timed out - I updated the timeout to 3 hours and the job passed. Here is an example of how to configure the timeout:

decoration_config:
grace_period: 5m0s
timeout: 4h0m0s

Once we have the timeout increased for this and the publish job - I think this looks good.

decoration_config:
grace_period: 5m0s
timeout: 4h0m0s
labels:
preset-dind-enabled: "true"
preset-docker-mirror-proxy: "true"
Expand All @@ -103,16 +106,16 @@ presubmits:
- "-ce"
- |
cd images
./publish_image.sh -b bootstrap quay.io kubevirtci
./publish_image.sh -b golang quay.io kubevirtci
./publish_multiarch_image.sh -b bootstrap quay.io kubevirtci
./publish_multiarch_image.sh -b -l golang quay.io kubevirtci
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
memory: "1Gi"
memory: "8Gi"
limits:
memory: "1Gi"
memory: "8Gi"
- name: build-kubekins-e2e-image
always_run: false
run_if_changed: "images/kubekins-e2e/.*"
Expand Down
3 changes: 2 additions & 1 deletion images/bootstrap/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ RUN mkdir /docker-graph
#

# Cache the most commonly used bazel versions in the container
RUN curl -Lo ./bazelisk https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64 && \
ARG ARCH
RUN curl -Lo ./bazelisk https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-${ARCH} && \
chmod +x ./bazelisk && mv ./bazelisk /usr/local/bin/bazelisk && \
cd /usr/local/bin && ln -s bazelisk bazel

Expand Down
3 changes: 2 additions & 1 deletion images/golang/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
FROM bootstrap
ARG ARCH
FROM bootstrap-$ARCH

ENV GIMME_GO_VERSION=1.17.8

Expand Down
121 changes: 121 additions & 0 deletions images/publish_multiarch_image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
#!/bin/bash -xe
archs=(amd64 arm64)

main() {
local build_only local_base_image
local_base_image=false
while getopts "blh" opt; do
case "$opt" in
b)
build_only=true
;;
l)
local_base_image=true
;;
h)
help
exit 0
;;
*)
echo "Invalid argument: $opt"
help
exit 1
esac
done
shift $((OPTIND-1))
local build_target="${1:?}"
local registry="${2:?}"
local registry_org="${3:?}"
local full_image_name image_tag base_image

image_tag="$(get_image_tag)"
full_image_name="$(
get_full_image_name \
"$registry" \
"$registry_org" \
"${build_target##*/}" \
"$image_tag"
)"

(
cd "$build_target"
base_image="$(get_base_image)"

build_image $local_base_image "$build_target" "$full_image_name" "$base_image"
)
[[ $build_only ]] && return
publish_image "$full_image_name"
publish_manifest "$full_image_name"
}

help() {
cat <<EOF
Usage:
./publish_multiarch_image.sh [OPTIONS] BUILD_TARGET REGISTRY REGISTRY_ORG

Build and publish multiarch infra images.

OPTIONS
-h Show this help message and exit.
-b Only build the image and exit. Do not publish the built image.
-l Use local base image
EOF
}

get_base_image() {
name="$(cat Dockerfile |grep FROM|awk '{print $2}')"
echo "${name}"
}

get_image_tag() {
local current_commit today
current_commit="$(git rev-parse HEAD)"
today="$(date +%Y%m%d)"
echo "v${today}-${current_commit:0:7}"
}

build_image() {
local local_base_image=${1:?}
local build_target="${2:?}"
local image_name="${3:?}"
local base_image="${4:?}"
# add qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
# build multi-arch images
for arch in ${archs[*]};do
if [[ $local_base_image == false ]]; then
brianmcarey marked this conversation as resolved.
Show resolved Hide resolved
docker pull --platform="linux/${arch}" ${base_image}
fi
docker build --platform="linux/${arch}" --build-arg ARCH=${arch} --build-arg IMAGE_ARG=${build_target} . -t "${image_name}-${arch}" -t "${build_target}-${arch}"
done
}

publish_image() {
local full_image_name="${1:?}"
for arch in ${archs[*]};do
docker push "${full_image_name}-${arch}"
done
}

publish_manifest() {
export DOCKER_CLI_EXPERIMENTAL="enabled"
local amend
local full_image_name="${1:?}"
amend=""
for arch in ${archs[*]};do
amend+=" --amend ${full_image_name}-${arch}"
done
docker manifest create ${full_image_name} ${amend}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be cool to try this with a tool like buildah but we can try doing that later. Other kubevirt projects are starting to adopt buildah for multi-arch builds - kubevirt/hostpath-provisioner#115

docker manifest push --purge ${full_image_name}
}

get_full_image_name() {
local registry="${1:?}"
local registry_org="${2:?}"
local image_name="${3:?}"
local tag="${4:?}"

echo "${registry}/${registry_org}/${image_name}:${tag}"
}

main "$@"