update argocd and jenkins in cd demo and script for minikube

examples/cicd-argocd/README.md

@@ -8,15 +8,17 @@
 This example uses the following components to setup a demo "gitops" pipleline that deploys a dummy ML model.
 
 * Jenkins
-* Argo
-* Argocd
+* [Argo](https://github.com/argoproj/argo)
+* [Argocd](https://github.com/argoproj/argo-cd)
 * Github (remote repos)
 * Git (local repos)
 * Local Docker Registry
 * Seldon Core
 
 ![missing cicd image](https://raw.githubusercontent.com/SeldonIO/seldon-core/master/examples/cicd-argocd/cicd-demo.png "Seldon Core CICD demo")
 
+Images are built by firing argo jobs from jenkins, justing a deployer image that contains kubectl to launch argo jobs. Models using the images are deployed by ArgoCD, which syncs a git repo to k8s.
+
 ## Setup
 
 * directory for the scripts
@@ -34,15 +36,16 @@ cd /some/path/
 git clone https://github.com/<your github id>/cicd-demo-model-source-files
 git clone https://github.com/<your github id>/cicd-demo-k8s-manifest-files
 ```
-* Create cluster, at least Kubernetes 10
+* Create cluster, at least Kubernetes 1.12. Otherwise point the model argo jobs at a different [seldonio/k8s-deployer tag](https://hub.docker.com/r/seldonio/k8s-deployer)
 
 * Have "helm", "argo", "argocd" installed
+* Shell needs to be able to run python, vim and tmux
 
 * (if gcp) create-user-cluster-admin-binding
 ```
 kubectl create clusterrolebinding user-cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)
 ```
-* Create "settings.sh" using "settings.sh.example"
+* Create "settings.sh" using "settings.sh.example". Be sure to set path to dir where projects cloned. Then run the script.
 
 * Install helm
 ```
@@ -52,7 +55,7 @@ kubectl create clusterrolebinding user-cluster-admin-binding --clusterrole=clust
 ```
 ./start-all
 ```
-* Create tmux windows (Not from inside another tmux session)
+* Create tmux windows (Not from inside another tmux session). Note at this point prediction call will fail as haven't deployed model yet.
 ```
 ./create-demo-tmux-session
 ```
@@ -68,24 +71,15 @@ kubectl create clusterrolebinding user-cluster-admin-binding --clusterrole=clust
 ```
 * Setup Jenkins
 ```
-# get initial browser login details, and use top login
+# open browser to jenkins login to create initial user
 ./jenkins/get-jenkins-browser-login
 
-IMPORTANT: fix any plugin issues, eg. update pipeline-job plugin if necessary and Reboot
-
-Install "Github" jenkins plugin
-Manage jenkins->Manage Plugins->Available
-    Find the "Github" plugin
-    Install without restart
+NOTE: if you see plugin issues, you can fix here or in the helm values file
 
-# setup security to use "Jenkins’ own user database"
-Manage Jenkins->Configure Global Security
-        - select "Jenkins’ own user database"
-        - Make sure "Allow users to sign up" is unchecked
-        - save
+On Jenkins login page you will need to create "Create First Admin User"
+            - set in the JENKINS_USER_NAME and JENKINS_USER_PASSWORD in the "settings.sh" file
 
-Jenkins will ask to "Create First Admin User"
-            - use the JENKINS_USER_NAME and JENKINS_USER_PASSWORD in the "settings.sh" file
+You need to have the user in the settings file as Jenkins needs that user in its store in order to issue a crumb for job creation. If you want you can disable user signup after creating the user.
 ```
 * Import Jenkins jobs
 ```
@@ -98,24 +92,29 @@ Jenkins will ask to "Create First Admin User"
 # get cmd-line login details, and use to login
 ./argocd/get-argocd-cmd-line-login
 
-# add current cluster to list
+# optionally add current cluster to list
+# likely not needed as current cluster should already be in list
 ./argocd/argocd-cluster-add
 ./argocd/argocd-cluster-list
 
 # create app
 ./argocd/argocd-app-create
 
 # get browser login details, and use to login
+# if minikube see argocd-cmd-line-login comment
 ./argocd/get-argocd-browser-login
 ```
 * Create Github Webhooks
 ```
 # For CI, add webhook to "cicd-demo-model-source-files" repo
 # get the webhook details to use
+# For this public hosting of Jenkins is necessary. If using minikube you can manually trigger or enable polling.
+# Registering the hook manually via github UI
 ./jenkins/get-jenkins-github-webhook-details
 
 # For CD, add webhook to "cicd-demo-k8s-manifest-files" repo
 # get the webhook details to use
+# For this public hosting of ArgoCD is necessary. If using minikube you can manually trigger or enable polling.
 ./argocd/get-argocd-github-webhook-details
 ```
 ## Operation
@@ -128,11 +127,11 @@ This session has a port forwarding window and a window that is a view on the ope
 Here the source of the dummy model can be committed and pushed to the remote Github repo. If the web hooks are setup it will trigger an auto build of the image.
 
 This creates a new docker image which is a new version of the model. Also the seldon deployment manifest is updated and pushed to the remote repo.  
-At this this point argocd will show the deployment is out sync. The new version of the model can now be manually deployed by getting argocd to 'sync' the updates.
+At this this point argocd will show the deployment is out sync. The new version of the model can now be manually deployed by getting argocd to 'sync' the updates (either from the argocd UI or with `argocd app sync cicd-demo-argocd`).
 
-The deployment of the new model will be seen as a rolling update in the session view.  Once the new model is ready the predictions will chnage to reflect the new version of the chnages.
+The deployment of the new model will be seen as a rolling update in the session view.  Once the new model is ready the predictions will change to reflect the new version of the changes.
 
-Argocd can also the set to 'auto-sync' the changes. This will automate the full pipeline.  
+Argocd can also be set to 'auto-sync' the changes (`argocd app set cicd-demo-argocd --sync-policy automated`). This will automate the full pipeline.
 Now changes to model that are committed and pushed will trigger the auto build and auto deploy the new version of the model. 
 
 ## Clean Up
@@ -149,5 +148,4 @@ Now changes to model that are committed and pushed will trigger the auto build a
 * Remove helm
 ```
 ./seldon-core/remove-helm
-```
-
+```

examples/cicd-argocd/argocd/argocd-app-create

@@ -17,6 +17,4 @@ argocd app create cicd-demo-argocd \
     --path demo-manifests \
     --dest-namespace default \
     --dest-server https://kubernetes.default.svc \
-    --sync-policy "${SYNC_POLICY}" \
-    --auto-prune
-
+    --sync-policy "${SYNC_POLICY}"

examples/cicd-argocd/argocd/get-argocd-browser-login

@@ -7,14 +7,14 @@ set -o pipefail
 STARTUP_DIR="$( cd "$( dirname "$0" )" && pwd )"
 
 source ${STARTUP_DIR}/../settings.sh
+source ${STARTUP_DIR}/get-argocd-server-url
 
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
-EXTERNAL_IP=$(${KUBECTL} get svc -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
-ADMIN_PASSWORD=$(${KUBECTL} get pods -n argocd -l app=argocd-server -o name | cut -d'/' -f 2)
+ADMIN_PASSWORD=$(${KUBECTL} get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2)
 
 echo ""
-echo "url: https://${EXTERNAL_IP}/"
+echo "url: https://"$(getUrl)
 echo "user: admin"
 echo "password: ${ADMIN_PASSWORD}"
 echo ""

examples/cicd-argocd/argocd/get-argocd-cmd-line-login

@@ -7,13 +7,20 @@ set -o pipefail
 STARTUP_DIR="$( cd "$( dirname "$0" )" && pwd )"
 
 source ${STARTUP_DIR}/../settings.sh
+source ${STARTUP_DIR}/get-argocd-server-url
 
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
-EXTERNAL_IP=$(${KUBECTL} get svc -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
-ADMIN_PASSWORD=$(${KUBECTL} get pods -n argocd -l app=argocd-server -o name | cut -d'/' -f 2)
+ADMIN_PASSWORD=$(${KUBECTL} get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2)
+
+login="argocd login "$(getUrl)" --username admin --password ${ADMIN_PASSWORD}"
+
+if [ -n "${KUBE_CONTEXT}" ]
+then
+  login+=" --name ${KUBE_CONTEXT}"
+fi
 
 echo ""
-echo "argocd login ${EXTERNAL_IP} --name ${KUBE_CONTEXT} --username admin --password ${ADMIN_PASSWORD}"
+echo $login
 echo ""
 

examples/cicd-argocd/argocd/get-argocd-server-url

@@ -10,7 +10,29 @@ source ${STARTUP_DIR}/../settings.sh
 
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
-EXTERNAL_IP=$(${KUBECTL} get svc -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+getUrl() {
+    EXTERNAL_IP=$(${KUBECTL} get svc -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+    NODEPORT=$(${KUBECTL} get svc -n argocd argocd-server -o jsonpath='{.spec.ports[1].nodePort}')
 
-echo "https://${EXTERNAL_IP}/"
+    url="argocd-url-unknown"
 
+    if [ -n "${EXTERNAL_IP}" ]
+    then
+          url="${EXTERNAL_IP}"
+    else
+          if [ -z "${KUBE_CONTEXT}" ]
+          then
+            KUBE_CONTEXT=$(kubectl config current-context)
+          fi
+          if [ "${KUBE_CONTEXT}" = "minikube" ]
+          then
+            EXTERNAL_IP=$(minikube ip)
+          else
+            EXTERNAL_IP="localhost"
+          fi
+          url="${EXTERNAL_IP}:${NODEPORT}"
+    fi
+    echo $url
+}
+
+echo $(getUrl)

examples/cicd-argocd/argocd/port-forward-argocd

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+set -o nounset
+set -o errexit
+set -o pipefail
+
+STARTUP_DIR="$( cd "$( dirname "$0" )" && pwd )"
+
+source ${STARTUP_DIR}/../settings.sh
+
+KUBECTL="kubectl --context=${KUBE_CONTEXT}"
+
+${KUBECTL} -n argocd port-forward deployment.apps/argocd-server 9091:8080

examples/cicd-argocd/argocd/show-argocd-logs

@@ -11,5 +11,5 @@ source ${STARTUP_DIR}/../settings.sh
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
 NAMESPACE_NAME=argocd
-${KUBECTL} logs -f $(kubectl --context="${KUBE_CONTEXT}" get pods -n ${NAMESPACE_NAME} -l app=argocd-server -o jsonpath='{.items[0].metadata.name}') -n ${NAMESPACE_NAME}
+${KUBECTL} logs -f $(kubectl --context="${KUBE_CONTEXT}" get pods -n ${NAMESPACE_NAME} -l app.kubernetes.io/name=argocd-server -o jsonpath='{.items[0].metadata.name}') -n ${NAMESPACE_NAME}
 

examples/cicd-argocd/docker-devbox-manifest-updater

@@ -8,7 +8,7 @@ STARTUP_DIR="$( cd "$( dirname "$0" )" && pwd )"
 
 source ${STARTUP_DIR}/settings.sh
 
-IMAGE_VERSION=k8s_v1.9.0
+IMAGE_VERSION=k8s_v1.14.0
 IMAGE_NAME="seldonio/k8s-deployer"
 
 docker run --rm -it \

examples/cicd-argocd/jenkins/get-jenkins-browser-login

@@ -7,17 +7,19 @@ set -o pipefail
 STARTUP_DIR="$( cd "$( dirname "$0" )" && pwd )"
 
 source ${STARTUP_DIR}/../settings.sh
+source ${STARTUP_DIR}/get-jenkins-url
 
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
 #printf $(${KUBECTL} get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
 
-EXTERNAL_IP=$(${KUBECTL} get svc -n jenkins jenkins -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
 ADMIN_PASSWORD=$(printf $(${KUBECTL} get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode))
 
 echo ""
-echo "url: http://${EXTERNAL_IP}:8080/"
-echo "user: admin"
-echo "password: ${ADMIN_PASSWORD}"
+echo "url: http://"$(getUrl)
+echo "setup: http://"$(getUrl)"signup"
+#don't echo initial password as using local store so user hs to create
+#echo "user: admin"
+#echo "password: ${ADMIN_PASSWORD}"
 echo ""
 

examples/cicd-argocd/jenkins/get-jenkins-url

@@ -10,7 +10,29 @@ source ${STARTUP_DIR}/../settings.sh
 
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
-EXTERNAL_IP=$(${KUBECTL} get svc -n jenkins jenkins -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+getUrl() {
+    EXTERNAL_IP=$(${KUBECTL} get svc -n jenkins jenkins -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+    NODEPORT=$(${KUBECTL} get svc -n jenkins jenkins -o jsonpath='{.spec.ports[0].nodePort}')
 
-echo "http://${EXTERNAL_IP}:8080/"
+    url="jenkins-url-unknown"
 
+    if [ -n "${EXTERNAL_IP}" ]
+    then
+          url="${EXTERNAL_IP}:8080/"
+    else
+          if [ -z "${KUBE_CONTEXT}" ]
+          then
+            KUBE_CONTEXT=$(kubectl config current-context)
+          fi
+          if [ "${KUBE_CONTEXT}" = "minikube" ]
+          then
+            EXTERNAL_IP=$(minikube ip)
+          else
+            EXTERNAL_IP="localhost"
+          fi
+          url="${EXTERNAL_IP}:${NODEPORT}/"
+    fi
+    echo $url
+}
+
+echo $(getUrl)

examples/cicd-argocd/jenkins/start-jenkins

@@ -12,15 +12,18 @@ KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
 #JENKINS_CHART_VERSION=0.15.1
 #JENKINS_CHART_VERSION=0.19.1
-JENKINS_CHART_VERSION=0.28.11
+JENKINS_CHART_VERSION=0.39.0
 
 ${KUBECTL} create namespace jenkins --dry-run -o yaml|${KUBECTL} apply -f -
 
 helm install --kube-context="${KUBE_CONTEXT}" stable/jenkins \
     --name jenkins \
     --namespace jenkins \
     --set rbac.install=true \
-    --version $JENKINS_CHART_VERSION
+    --version $JENKINS_CHART_VERSION \
+    --set Master.AdminUser=$JENKINS_USER_NAME \
+    --set Master.AdminPassword=$JENKINS_USER_PASSWORD \
+    --values ${STARTUP_DIR}/values.yml
 
 # create a rolebing for jenkins
 ${KUBECTL} create rolebinding jenkins-admin --clusterrole=admin --serviceaccount=jenkins:default --namespace=default --dry-run -o yaml | \

examples/cicd-argocd/jenkins/values.yml

@@ -0,0 +1,11 @@
+Master:
+  InstallPlugins:
+    - kubernetes:1.15.1
+    - workflow-job:2.32
+    - workflow-aggregator:2.6
+    - credentials-binding:1.18
+    - git:3.9.3
+    - github:1.29.3
+#local jenkins store needed for crumb
+  SecurityRealm: |-
+    <securityRealm class="hudson.security.HudsonPrivateSecurityRealm"/>

examples/cicd-argocd/k8s-local-docker-registry/start-docker-private-registry

@@ -11,6 +11,6 @@ source ${STARTUP_DIR}/../settings.sh
 KUBECTL="kubectl --context=${KUBE_CONTEXT}"
 
 ${KUBECTL} apply -f ${STARTUP_DIR}/docker-private-registry.json -n default
-${KUBECTL} rollout status deploy/docker-private-registry-deployment
+${KUBECTL} rollout status deploy/docker-private-registry-deployment -n default
 ${KUBECTL} apply -f ${STARTUP_DIR}/docker-private-registry-proxy.json -n default
 

examples/cicd-argocd/settings.sh.example

@@ -7,8 +7,8 @@ K8S_MANIFEST_FILES_DIR=/pathto/cicd-demo-k8s-manifest-files
 GITHUB_USER=<github user id>
 GITHUB_TOKEN=<github personal token>
 
-ARGOCD_VERSION=v0.10.6
+ARGOCD_VERSION=v0.12.0
 
-JENKINS_USER_NAME=<jenkins user id>
-JENKINS_USER_PASSWORD=<jenkins password>
+JENKINS_USER_NAME=
+JENKINS_USER_PASSWORD=