Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build-make-check-container job #1957

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions build-make-check-container/build/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# This is just the default. Overridden by `docker build --build-arg DISTRO=$DISTRO` at build time
ARG DISTRO='ubuntu:20.04'
FROM $DISTRO
RUN if command -v yum; then \
yum install -y epel-release; \
yum install -y git sudo ccache clang; \
yum update -y; \
else \
apt update && DEBIAN_FRONTEND="noninteractive" apt install -y git-core sudo ccache; \
fi
RUN git clone https://github.com/ceph/ceph /opt/ceph
RUN git clone https://github.com/ceph/ceph-ci /opt/ceph-ci
# Set some defaults (again, overridden using --build-arg)
ARG INSTALL_DEPS_REPO=ceph
ARG INSTALL_DEPS_BRANCH=master
RUN cd /opt/$INSTALL_DEPS_REPO \
&& if ! git checkout $INSTALL_DEPS_BRANCH; then \
echo $INSTALL_DEPS_BRANCH doesn\'t exist; \
exit 1; \
fi \
&& DEBIAN_FRONTEND="noninteractive" ./install-deps.sh \
&& git submodule update --init --recursive
Comment on lines +1 to +22
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would try something like that:

Suggested change
# This is just the default. Overridden by `docker build --build-arg DISTRO=$DISTRO` at build time
ARG DISTRO='ubuntu:20.04'
FROM $DISTRO
RUN if command -v yum; then \
yum install -y epel-release; \
yum install -y git sudo ccache clang; \
yum update -y; \
else \
apt update && DEBIAN_FRONTEND="noninteractive" apt install -y git-core sudo ccache; \
fi
RUN git clone https://github.com/ceph/ceph /opt/ceph
RUN git clone https://github.com/ceph/ceph-ci /opt/ceph-ci
# Set some defaults (again, overridden using --build-arg)
ARG INSTALL_DEPS_REPO=ceph
ARG INSTALL_DEPS_BRANCH=master
RUN cd /opt/$INSTALL_DEPS_REPO \
&& if ! git checkout $INSTALL_DEPS_BRANCH; then \
echo $INSTALL_DEPS_BRANCH doesn\'t exist; \
exit 1; \
fi \
&& DEBIAN_FRONTEND="noninteractive" ./install-deps.sh \
&& git submodule update --init --recursive
# This is just the default. Overridden by `docker build --build-arg DISTRO=$DISTRO` at build time
ARG DISTRO='ubuntu:20.04'
# Set some defaults (again, overridden using --build-arg)
ARG INSTALL_DEPS_REPO=ceph
ARG INSTALL_DEPS_BRANCH=master
FROM $DISTRO
RUN command -v yum && (yum install -y epel-release && yum install -y git sudo ccache clang && yum update -y) || (apt update && DEBIAN_FRONTEND="noninteractive" apt install -y git-core sudo ccache) && \
git clone -b $INSTALL_DEPS_BRANCH --depth 1 https://github.com/ceph/ceph /opt/ceph || (echo "$INSTALL_DEPS_BRANCH doesn't exist" && exit 1) && git clone https://github.com/ceph/ceph-ci /opt/ceph-ci && \
cd /opt/$INSTALL_DEPS_REPO && DEBIAN_FRONTEND="noninteractive" ./install-deps.sh && git submodule update --init --recursive

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

less steps, less layers, more readable...
just saying... 🤷‍♂️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will admit this is my first Dockerfile but I find yours to be far less readable.

Using mine, if one of the RUN steps fails, you can tell where and why right away vs. having to wait for the entire single RUN step in your suggestion. I don't want to end up with another one of these:

STEP 16/16: RUN     yum install -y epel-release https://apache.jfrog.io/artifactory/arrow/centos/8/apache-arrow-release-latest.rpm && yum install -y jq && bash -c '   if [ -n "nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rgw nfs-ganesha-rados-grace nfs-ganesha-rados-urls" ]; then     echo "[ganesha]" > /etc/yum.repos.d/ganesha.repo ;     echo "name=ganesha" >> /etc/yum.repos.d/ganesha.repo ;     if [[ "${CEPH_VERSION}" == master ]]; then       echo "baseurl=[https://buildlogs.centos.org/centos/\$releasever/storage/\$basearch/nfsganesha-3/](https://buildlogs.centos.org/centos//$releasever/storage//$basearch/nfsganesha-3/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=0" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;     elif [[ "${CEPH_VERSION}" == pacific ]]; then       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V3.5-stable/$CEPH_VERSION/el\$releasever/\$basearch/](https://download.ceph.com/nfs-ganesha/rpm-V3.5-stable/$CEPH_VERSION/el/$releasever//$basearch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;       echo "[ganesha-noarch]" >> /etc/yum.repos.d/ganesha.repo ;       echo "name=ganesha-noarch" >> /etc/yum.repos.d/ganesha.repo ;       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V3.5-stable/$CEPH_VERSION/el\$releasever/noarch/](https://download.ceph.com/nfs-ganesha/rpm-V3.5-stable/$CEPH_VERSION/el/$releasever/noarch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;     elif [[ "${CEPH_VERSION}" == octopus ]]; then       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V3.3-stable/$CEPH_VERSION/el\$releasever/\$basearch/](https://download.ceph.com/nfs-ganesha/rpm-V3.3-stable/$CEPH_VERSION/el/$releasever//$basearch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;       echo "[ganesha-noarch]" >> /etc/yum.repos.d/ganesha.repo ;       echo "name=ganesha-noarch" >> /etc/yum.repos.d/ganesha.repo ;       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V3.3-stable/$CEPH_VERSION/el\$releasever/noarch/](https://download.ceph.com/nfs-ganesha/rpm-V3.3-stable/$CEPH_VERSION/el/$releasever/noarch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;     elif [[ "${CEPH_VERSION}" == nautilus ]]; then       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V2.8-stable/$CEPH_VERSION/\$basearch/](https://download.ceph.com/nfs-ganesha/rpm-V2.8-stable/$CEPH_VERSION//$basearch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;     else       echo "baseurl=[https://download.ceph.com/nfs-ganesha/rpm-V2.7-stable/$CEPH_VERSION/\$basearch/](https://download.ceph.com/nfs-ganesha/rpm-V2.7-stable/$CEPH_VERSION//$basearch/)" >> /etc/yum.repos.d/ganesha.repo ;       echo "gpgcheck=1" >> /etc/yum.repos.d/ganesha.repo ;       echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo  ;     fi ;   fi ;   if [ -n "tcmu-runner ceph-iscsi python3-rtslib" ]; then     curl -s -L https://shaman.ceph.com/api/repos/tcmu-runner/master/latest/centos/8/repo?arch=$(arch) -o /etc/yum.repos.d/tcmu-runner.repo ;     if [[ "${CEPH_VERSION}" =~ master ]]; then       curl -s -L https://shaman.ceph.com/api/repos/ceph-iscsi/master/latest/centos/8/repo -o /etc/yum.repos.d/ceph-iscsi.repo ;     elif [[ "${CEPH_VERSION}" =~ nautilus|octopus|pacific ]]; then       curl -s -L https://download.ceph.com/ceph-iscsi/3/rpm/el8/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ;     else       curl -s -L https://download.ceph.com/ceph-iscsi/2/rpm/el8/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ;     fi ;   fi' && yum update -y --setopt=install_weak_deps=False && rpm --import 'https://download.ceph.com/keys/release.asc' && if [[ "${CEPH_VERSION}" == nautilus ]]; then   CEPH_MGR_K8SEVENTS="ceph-mgr-k8sevents";   if [[ -n "" ]]; then     CPR= ;     if [[ ${CPR:1:2} -eq 14 ]] && [[ ${CPR:4:1} -eq 2 ]] && [[ ${CPR:6} -lt 5 ]]; then       CEPH_MGR_K8SEVENTS="" ;     fi ;   fi ; fi && bash -c '   if [[ "${CEPH_VERSION}" =~ master ]] || ${CEPH_DEVEL}; then     ARCH=$(arch);     if [[ "${ARCH}" == "aarch64" ]]; then       ARCH="arm64";     fi ;     REPO_URL=$(curl -s "https://shaman.ceph.com/api/search/?project=ceph&distros=centos/8/${ARCH}&flavor=${OSD_FLAVOR}&ref=${CEPH_REF}&sha1=latest" | jq -r .[0].url);     RELEASE_VER=0 ;    if [[ "${OSD_FLAVOR}" == "crimson" ]]; then      CRIMSON_PACKAGES="ceph-crimson-osd";    fi ;  else     RELEASE_VER=1 ;    REPO_URL="[http://download.ceph.com/rpm-${CEPH_VERSION}/el8/](http://download.ceph.com/rpm-$%7BCEPH_VERSION%7D/el8/)";   fi &&   rpm -Uvh "$REPO_URL/noarch/ceph-release-1-${RELEASE_VER}.el8.noarch.rpm" &&   if [[ 8 -eq 8 ]]; then     yum install -y dnf-plugins-core ;     yum copr enable -y tchaikov/python-scikit-learn ;     yum copr enable -y tchaikov/python3-asyncssh ;   fi ' && yum install -y --setopt=install_weak_deps=False --enablerepo=powertools         ca-certificates         e2fsprogs         ceph-common          ceph-mon          ceph-osd         ceph-mds cephfs-mirror         rbd-mirror          ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook python3-saml        ceph-grafana-dashboards         kmod         lvm2         gdisk         smartmontools         nvme-cli         libstoragemgmt         systemd-udev         ceph-radosgw libradosstriper1         nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rgw nfs-ganesha-rados-grace nfs-ganesha-rados-urls         tcmu-runner ceph-iscsi python3-rtslib         attr ceph-fuse rbd-nbd         ${CRIMSON_PACKAGES}         ceph-immutable-object-cache         python3-scikit-learn         ceph-volume &&     INITIAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')" &&     echo 'Postinstall cleanup' &&  (rm -rf "/usr/bin/hyperkube /usr/bin/etcd /usr/bin/systemd-analyze /usr/share/hwdata/{iab.txt,oui.txt} /etc/profile.d/lang.sh" &&    yum clean all &&    rpm -q         ca-certificates         e2fsprogs         ceph-common          ceph-mon          ceph-osd         ceph-mds cephfs-mirror         rbd-mirror          ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook python3-saml        ceph-grafana-dashboards         kmod         lvm2         gdisk         smartmontools         nvme-cli         libstoragemgmt         systemd-udev         ceph-radosgw libradosstriper1         nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rgw nfs-ganesha-rados-grace nfs-ganesha-rados-urls         tcmu-runner ceph-iscsi python3-rtslib         attr ceph-fuse rbd-nbd         ${CRIMSON_PACKAGES}         ceph-immutable-object-cache         python3-scikit-learn         ceph-volume ) && sed -i -e 's/udev_rules = 1/udev_rules = 0/' -e 's/udev_sync = 1/udev_sync = 0/' -e 's/obtain_device_list_from_udev = 1/obtain_device_list_from_udev = 0/' /etc/lvm/lvm.conf && grep -sqo "udev_sync = 0" /etc/lvm/lvm.conf && grep -sqo "udev_rules = 0" /etc/lvm/lvm.conf && grep -sqo "obtain_device_list_from_udev = 0" /etc/lvm/lvm.conf &&     rm -rf         /etc/{selinux,systemd,udev}         /lib/{lsb,udev}         /tmp/*         /usr/lib{,64}/{locale,udev,dracut}         /usr/share/{doc,info,locale,man}         /usr/share/{bash-completion,pkgconfig/bash-completion.pc}         /var/log/*         /var/tmp/* &&     find  / -xdev -name "*.pyc" -o -name "*.pyo" -exec rm -f {} \; &&     if [ -f /usr/bin/ceph-dencoder ]; then gzip -9 /usr/bin/ceph-dencoder; fi &&     bash -c '       function ifstrip () { if compgen -g "$1"; then strip -s "$1"; fi } &&       ifstrip /usr/lib{,64}/ceph/erasure-code/* &&       ifstrip /usr/lib{,64}/rados-classes/* &&       ifstrip /usr/lib{,64}/python*/{dist,site}-packages/{rados,rbd,rgw}.*.so &&       ifstrip /usr/bin/{crushtool,monmaptool,osdmaptool}' &&     rm -f /usr/lib/ceph/mgr/dashboard/static/AdminLTE-*/plugins/datatables/extensions/TableTools/images/psd/* &&     find /var/log/ -type f -exec truncate -s 0 {} \; &&     FINAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')" &&     REMOVED_SIZE=$((INITIAL_SIZE - FINAL_SIZE)) &&     echo "Cleaning process removed ${REMOVED_SIZE}MB" &&     echo "Dropped container size from ${INITIAL_SIZE}MB to ${FINAL_SIZE}MB" &&     rpm -q         ca-certificates         e2fsprogs         ceph-common          ceph-mon          ceph-osd         ceph-mds cephfs-mirror         rbd-mirror          ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook python3-saml        ceph-grafana-dashboards         kmod         lvm2         gdisk         smartmontools         nvme-cli         libstoragemgmt         systemd-udev         ceph-radosgw libradosstriper1         nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rgw nfs-ganesha-rados-grace nfs-ganesha-rados-urls         tcmu-runner ceph-iscsi python3-rtslib         attr ceph-fuse rbd-nbd         ${CRIMSON_PACKAGES}         ceph-immutable-object-cache         python3-scikit-learn         ceph-volume && echo 'Packages verified successfully'

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although there's not only the readability concern here, I was just suggesting, your call then... 🤷‍♂️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to being educated 😄. What do less steps and layers mean?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each operation (RUN, COPY, ...) adds a new layer, the more layer you have, the bigger your image size will be.

You might want to have a look at "multi-stage" builds.

https://docs.docker.com/develop/develop-images/multistage-build/

# Doesn't work yet
# # To populate ccache
# && ./src/script/run-make.sh \
# && rm -rf build
# Some defaults. Overridden by `docker run -e VAR=$VAR` at runtime
ENV REPO=${REPO:-ceph}
ENV BRANCH=${BRANCH:-master}
ENV PULLREQUEST_ID=${PULLREQUEST_ID}
# Uncertain on the proper order here. Do I need to `pull` after checking out the branch again?
CMD cd /opt/$REPO \
&& git fetch --all \
&& if [ "$PULLREQUEST_ID" == "" ]; then \
echo "PULLREQUEST_ID was not defined. Using the \$BRANCH var"; \
echo -e "****************\n\nChecking out branch: $BRANCH\n\n****************"; \
git checkout $BRANCH; \
git pull; \
else \
echo -e "****************\n\nChecking out Pull Request ID: $PULLREQUEST_ID\n\n****************"; \
git fetch origin pull/$PULLREQUEST_ID/head:PR$PULLREQUEST_ID; \
git checkout PR$PULLREQUEST_ID; \
fi \
&& ./install-deps.sh \
&& timeout 3h ./run-make-check.sh
20 changes: 20 additions & 0 deletions build-make-check-container/build/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/bash
set -e

# Uncomment these to run locally on your workstation
#DISTRO="ubuntu:20.04"
#DISTRO="centos:8"
#BRANCH=${BRANCH:-master}
#WORKSPACE=~/git/ceph/ceph-build

DASHED_DISTRO=$(echo "$DISTRO" | sed 's/:/-/')

cd "$WORKSPACE"/build-make-check-container/build

sudo docker build --build-arg INSTALL_DEPS_BRANCH=$BRANCH --build-arg DISTRO=$DISTRO -t make-check:$BRANCH-$DASHED_DISTRO .

sudo docker login -u "${REGISTRY_USERNAME}" -p "${REGISTRY_PASSWORD}" quay.ceph.io

sudo docker image tag make-check:$BRANCH-$DASHED_DISTRO quay.ceph.io/cephci/make-check:$BRANCH-$DASHED_DISTRO

sudo docker push quay.ceph.io/cephci/make-check:$BRANCH-$DASHED_DISTRO
29 changes: 29 additions & 0 deletions build-make-check-container/build/setup
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env bash

set -ex

if [[ $(command -v docker) == '' ]]; then
# Set up docker official repo and install docker.
sudo apt update -y
sudo apt install \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update -y
sudo apt install -y docker-ce docker-ce-cli containerd.io
fi
sudo groupadd docker || true
sudo usermod -aG docker $(id -un)
sudo systemctl start docker
sudo chgrp "$(id -un)" /var/run/docker.sock

docker info
docker container prune -f


Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
- job:
name: build-make-check-container
node: master
project-type: matrix
defaults: global
display-name: 'Builds the ceph-make-check container and pushes to quay.ceph.io'
quiet-period: 5
block-downstream: false
block-upstream: false
retry-count: 3
execution-strategy:
combination-filter: |
!(BRANCH=="master" && DISTRO=="centos:7") &&
!(BRANCH=="pacific" && DISTRO=="centos:7") &&
!(BRANCH=="master" && DISTRO=="ubuntu:18.04") &&
!(BRANCH=="pacific" && DISTRO=="ubuntu:18.04")
properties:
- build-discarder:
days-to-keep: 14

# parameters:
# - string:
# name: BRANCH
# description: "Base branch you want to the container to have install-deps run for"
# default: master
#
# - string:
# name: DISTRO
# description: "Distro you want install-deps run on"
# default: 'ubuntu:20.04'

axes:
- axis:
type: label-expression
name: MACHINE_SIZE
values:
- huge
- axis:
type: label-expression
name: MACHINE_TYPE
values:
- focal
- axis:
type: label-expression
name: MACHINE_ARCH
values:
- x86_64

- axis:
type: user-defined
name: BRANCH
values:
- master
- octopus
- pacific

- axis:
type: user-defined
name: DISTRO
values:
- 'centos:7'
- 'centos:8'
- 'ubuntu:18.04'
- 'ubuntu:20.04'

triggers:
- timed: '@daily'

scm:
- git:
url: https://github.com/ceph/ceph-build.git
branches:
- build-make-check-container
browser: auto
timeout: 20

builders:
- shell:
!include-raw:
# - ../../../scripts/build_utils.sh
- ../../build/setup
- ../../build/build

wrappers:
- inject-passwords:
global: true
mask-password-params: true
- credentials-binding:
- username-password-separated:
credential-id: quay-ceph-io-make-check-bot
username: REGISTRY_USERNAME
password: REGISTRY_PASSWORD