Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support universal base image #95

Merged
merged 32 commits into from
Jun 25, 2020
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
64006e4
baby steps
sandoichi May 21, 2020
153c8f3
build out some more steps
sandoichi May 21, 2020
34922cb
write file with base os
sandoichi May 26, 2020
d562174
add logic to select ubi image in mage targets
sandoichi May 26, 2020
c7a9757
use ubi instead of busybox when necessary for logging container; buil…
sandoichi May 27, 2020
6db60cf
unify gh workflows and simplify steps
sandoichi May 27, 2020
a7da497
fixup! unify gh workflows and simplify steps
sandoichi May 27, 2020
8aac13d
fixup! fixup! unify gh workflows and simplify steps
sandoichi May 27, 2020
bb6dc8a
add debug print
sandoichi May 27, 2020
c7f6fd4
fixup workflow tags to push
sandoichi May 27, 2020
55e0cff
trimspace on gh package tags
sandoichi May 27, 2020
6236cd7
fixup! trimspace on gh package tags
sandoichi May 27, 2020
c4e6295
more trimspace
sandoichi May 27, 2020
b75fb46
add in logic for choosing ubi based config builder image
sandoichi May 29, 2020
97e30c2
add in dse and config builder ubi coords
sandoichi Jun 5, 2020
89540fd
add oss c* 3.11.6 image
sandoichi Jun 16, 2020
14d4c91
prefix ioutil import
sandoichi Jun 16, 2020
288402c
Add debug cat cmd to gh action
sandoichi Jun 16, 2020
7c7cfb8
Merge remote-tracking branch 'origin/master' into support-universal-b…
sandoichi Jun 16, 2020
ce01958
add conditions around checkout rules in gh workflow to avoid merging
sandoichi Jun 16, 2020
0710032
Remove debug print in gh workflow
sandoichi Jun 16, 2020
472a65e
Merge remote-tracking branch 'origin/master' into support-universal-b…
sandoichi Jun 24, 2020
3c6c780
use separate dockerfile for ubi images
sandoichi Jun 25, 2020
cb76c87
only run docker build once now
sandoichi Jun 25, 2020
7808c31
debug print in workflow
sandoichi Jun 25, 2020
ab4f8b6
more debug
sandoichi Jun 25, 2020
0aaf87b
fixup! more debug
sandoichi Jun 25, 2020
0b385eb
fixup! fixup! more debug
sandoichi Jun 25, 2020
96080eb
fixup! fixup! fixup! more debug
sandoichi Jun 25, 2020
c1ff59c
use env constant instead of string; more gh workflow debugging
sandoichi Jun 25, 2020
44fb640
workflow should be in good shape now
sandoichi Jun 25, 2020
200bff9
use correct docker tags file for gh package pushing..
sandoichi Jun 25, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
name: Cass Operator Build & Deploy
on: pull_request
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build_operator_docker:
name: Build Cass Operator Docker Image
Expand All @@ -9,14 +13,18 @@ jobs:
GOROOT: /usr/local/go1.13
steps:
- uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Go 1.13
uses: actions/setup-go@v1
with:
go-version: 1.13
- name: Install Mage
run: |
echo "GH_EVENT: [${GITHUB_EVENT_NAME}]"
cd /tmp
wget https://github.com/magefile/mage/releases/download/v1.9.0/mage_1.9.0_Linux-64bit.tar.gz
tar -xvf mage_1.9.0_Linux-64bit.tar.gz
Expand All @@ -31,14 +39,17 @@ jobs:
run: |
export PATH=$GOROOT/bin:$GOPATH/bin:$PATH
mage operator:testGenerateClient
- name: Build docker
- name: Build docker - standard and ubi images
env:
MO_BRANCH: ${{ github.event.pull_request.head.ref }}
PR_REF: ${{ github.event.pull_request.head.ref }}
MO_BASE_OS: 'registry.access.redhat.com/ubi7/ubi-minimal:7.8'
run: |
if [ ${GITHUB_EVENT_NAME} -eq "pull_request" ]; then export MO_BRANCH=${PR_REF}; fi;
echo "MO_BRANCH: [${MO_BRANCH}]"
export PATH=$GOROOT/bin:$GOPATH/bin:$PATH
mage operator:testAndBuild
- name: Deploy to ECR
if: github.event.pull_request.head.repo.full_name == 'datastax/cass-operator'
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'datastax/cass-operator'
env:
MO_ECR_ID: ${{ secrets.ECR_ID }}
MO_ECR_SECRET: ${{ secrets.ECR_SECRET }}
Expand All @@ -48,12 +59,12 @@ jobs:
export MO_TAGS=$(cat ./build/tagsToPush.txt)
mage operator:deployToECR
- name: Deploy to GH Packages
if: github.event.pull_request.head.repo.full_name == 'datastax/cass-operator'
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'datastax/cass-operator'
env:
MO_GH_USR: 'datastax/cass-operator'
MO_GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
MO_GH_PKG_REPO: 'datastax/cass-operator/operator'
run: |
export PATH=$GOROOT/bin:$GOPATH/bin:$PATH
export MO_TAGS=$(cat ./build/tagsToPush.txt)
export MO_TAGS=$(cat ./build/fullTagsToPush.txt)
mage operator:deployToGHPackages
56 changes: 0 additions & 56 deletions .github/workflows/operatorStableBuildAndDeploy.yml

This file was deleted.

20 changes: 17 additions & 3 deletions mage/k8s/lib.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (

const (
OperatorImage = "datastax/cass-operator:latest"
OperatorImageUBI = "datastax/cass-operator:latest-ubi"
envLoadDevImages = "M_LOAD_DEV_IMAGES"
envK8sFlavor = "M_K8S_FLAVOR"
)
Expand All @@ -38,9 +39,21 @@ var supportedFlavors = map[string]ClusterActions{
"gke": gcp.ClusterActions,
}

func getOperatorImage() string {
var img string
if baseOs := os.Getenv(operator.EnvBaseOs); baseOs != "" {
img = "datastax/cass-operator:latest-ubi"
} else {
img = "datastax/cass-operator:latest"
}
return img
}

func loadImagesFromBuildSettings(cfg ClusterActions, settings BuildSettings) {
for _, image := range settings.Dev.Images {
shutil.RunVPanic("docker", "pull", image)
// we likely don't always care if we fail to pull
// because we could be testing local images
_ = shutil.RunV("docker", "pull", image)
cfg.LoadImage(image)
}
}
Expand Down Expand Up @@ -96,7 +109,8 @@ func SetupEmptyCluster() {
clusterActions.ApplyDefaultStorage()
//TODO make this part optional
operator.BuildDocker()
clusterActions.LoadImage(OperatorImage)
operatorImg := getOperatorImage()
clusterActions.LoadImage(operatorImg)
}

// Bootstrap a cluster, then run Ginkgo integration tests.
Expand Down Expand Up @@ -131,8 +145,8 @@ func SetupExampleCluster() {
mg.Deps(SetupEmptyCluster)
kubectl.CreateSecretLiteral("cassandra-superuser-secret", "devuser", "devpass").ExecVPanic()

overrides := map[string]string{"image": getOperatorImage()}
var namespace = "default"
var overrides = map[string]string{"image": "datastax/cass-operator:latest"}
err := helm_util.Install("./charts/cass-operator-chart", "cass-operator", namespace, overrides)
mageutil.PanicOnError(err)

Expand Down
5 changes: 3 additions & 2 deletions mage/operator/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,10 @@ func retagAndPush(tags []string, remoteUrl string) {
func retagAndPushForGH(tags []string) {
pkgRepo := mageutil.RequireEnv(envGHPackageRepo)
reg := regexp.MustCompile(`.*\:`)
for _, tag := range tags {
for _, t := range tags {
tag := strings.TrimSpace(t)
updatedTag := reg.ReplaceAllString(tag, fmt.Sprintf("%s:", pkgRepo))
fullGHTag := fmt.Sprintf("%s/%s", ghPackagesRegistry, updatedTag)
fullGHTag := fmt.Sprintf("%s/%s", ghPackagesRegistry, strings.TrimSpace(updatedTag))
dockerTag(tag, fullGHTag)
fmt.Printf("- Pushing image %s\n", fullGHTag)
dockerutil.Push(fullGHTag).WithCfg(rootBuildDir).ExecVPanic()
Expand Down
54 changes: 43 additions & 11 deletions mage/operator/lib.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ import (
)

const (
dockerBase = "./operator/docker/base/Dockerfile"
dockerUbi = "./operator/docker/ubi/Dockerfile"
rootBuildDir = "./build"
sdkBuildDir = "operator/build"
diagramsDir = "./docs/developer/diagrams"
Expand All @@ -36,6 +38,7 @@ const (
envGitBranch = "MO_BRANCH"
envVersionString = "MO_VERSION"
envGitHash = "MO_HASH"
EnvBaseOs = "MO_BASE_OS"

errorUnstagedPreGenerate = `
Unstaged changes detected.
Expand Down Expand Up @@ -73,7 +76,7 @@ func checkForUnstagedChanges(message string) {
func writeBuildFile(fileName string, contents string) {
mageutil.EnsureDir(rootBuildDir)
outputPath := filepath.Join(rootBuildDir, fileName)
err := ioutil.WriteFile(outputPath, []byte(contents+"\n"), 0666)
err := ioutil.WriteFile(outputPath, []byte(contents), 0666)
if err != nil {
fmt.Printf("Failed to write file at %s\n", outputPath)
panic(err)
Expand Down Expand Up @@ -349,17 +352,34 @@ func calcFullVersion(settings cfgutil.BuildSettings, git GitData) FullVersion {
}
}

func runDockerBuild(version FullVersion) []string {
func calcVersionAndTags(version FullVersion, ubiBase bool) (string, []string) {
repoPath := "datastax/cass-operator"
versionedTag := fmt.Sprintf("%s:%v", repoPath, version)
tagsToPush := []string{
versionedTag,
fmt.Sprintf("%s:%s", repoPath, version.Hash),
var versionedTag string
var tagsToPush []string

if ubiBase {
versionedTag = fmt.Sprintf("%s:%v-ubi", repoPath, version)
tagsToPush = []string{
versionedTag,
fmt.Sprintf("%s:%s-ubi", repoPath, version.Hash),
fmt.Sprintf("%s:latest-ubi", repoPath),
}
} else {
versionedTag = fmt.Sprintf("%s:%v", repoPath, version)
tagsToPush = []string{
versionedTag,
fmt.Sprintf("%s:%s", repoPath, version.Hash),
fmt.Sprintf("%s:latest", repoPath),
}
}
tags := append(tagsToPush, fmt.Sprintf("%s:latest", repoPath))

return versionedTag, tagsToPush
}

func runDockerBuild(versionedTag string, dockerTags []string, extraBuildArgs []string, dockerfile string) {
buildArgs := []string{fmt.Sprintf("VERSION_STAMP=%s", versionedTag)}
dockerutil.Build(".", "", "./operator/Dockerfile", tags, buildArgs).ExecVPanic()
return tagsToPush
buildArgs = append(buildArgs, extraBuildArgs...)
dockerutil.Build(".", "", dockerfile, dockerTags, buildArgs).ExecVPanic()
}

func runGoBuild(version string) {
Expand Down Expand Up @@ -423,12 +443,24 @@ func BuildDocker() {
settings := cfgutil.ReadBuildSettings()
git := getGitData()
version := calcFullVersion(settings, git)
operatorTags := runDockerBuild(version)

//build regular docker image
versionedTag, dockerTags := calcVersionAndTags(version, false)
runDockerBuild(versionedTag, dockerTags, nil, dockerBase)

if baseOs := os.Getenv(EnvBaseOs); baseOs != "" {
//build ubi docker image
args := []string{fmt.Sprintf("BASE_OS=%s", baseOs)}
ubiVersionedTag, ubiDockerTags := calcVersionAndTags(version, true)
runDockerBuild(ubiVersionedTag, ubiDockerTags, args, dockerUbi)
dockerTags = append(dockerTags, ubiDockerTags...)
}

// Write the versioned image tags to a file in our build
// directory so that other targets in the build process can identify
// what was built. This is particularly important to know
// for targets that retag and deploy to external docker repositories
outputText := strings.Join(operatorTags, "|")
outputText := strings.Join(dockerTags, "|")
writeBuildFile("tagsToPush.txt", outputText)
}

Expand Down
36 changes: 36 additions & 0 deletions operator/cmd/manager/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"errors"
"flag"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"runtime"
Expand Down Expand Up @@ -120,6 +121,7 @@ func main() {
ctx := context.Background()
// Become the leader before proceeding
err = leader.Become(ctx, "cass-operator-lock")

if err != nil {
log.Error(err, "could not become leader")
os.Exit(1)
Expand All @@ -132,6 +134,10 @@ func main() {
log.Error(err, "Failed to ensure webhook CA configuration")
}

if err = readBaseOsIntoEnv(); err != nil {
log.Error(err, "Failed to read base OS into env")
}

// Set default manager options
options := manager.Options{
Namespace: namespace,
Expand Down Expand Up @@ -200,6 +206,36 @@ func main() {
}
}

func readBaseOsIntoEnv() error {
baseOsArgFilePath := "/var/lib/cass-operator/base_os"

info, err := os.Stat(baseOsArgFilePath)
if os.IsNotExist(err) {
msg := fmt.Sprintf("Could not locate base OS arg file at %s", baseOsArgFilePath)
err = fmt.Errorf("%s. %v", msg, err)
return err
}

if info.IsDir() {
msg := fmt.Sprintf("Base OS arg path is a directory not a file: %s", baseOsArgFilePath)
err = fmt.Errorf("%s. %v", msg, err)
return err
}

rawVal, err := ioutil.ReadFile(baseOsArgFilePath)
if err != nil {
msg := fmt.Sprintf("Failed to read base OS arg file at %s", baseOsArgFilePath)
err = fmt.Errorf("%s. %v", msg, err)
return err
}

baseOs := strings.TrimSpace(string(rawVal))
os.Setenv("BASE_IMAGE_OS", baseOs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason not to use the EnvBaseImageOs constant here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, good catch!

log.Info(fmt.Sprintf("BASE_IMAGE_OS set to '%s'", baseOs))

return nil
}

// addMetrics will create the Services and Service Monitors to allow the operator export the metrics by using
// the Prometheus operator
func addMetrics(ctx context.Context, cfg *rest.Config) {
Expand Down
3 changes: 3 additions & 0 deletions operator/Dockerfile → operator/docker/base/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ RUN MO_VERSION=${VERSION_STAMP} mage operator:buildGo
FROM alpine:3.9

ENV GOPATH=/go

RUN mkdir -p /var/lib/cass-operator/
RUN touch /var/lib/cass-operator/base_os
WORKDIR /go

# All we need from the builder image is operator executable
Expand Down
Loading