Merge branch 'master' into pod-priority

docs/using_lagoon/drupal/lagoonize.md

@@ -2,9 +2,9 @@
 
 ## 1. Lagoon Drupal Setting Files
 
-In order for Drupal to work with Lagoon we need to teach Drupal about Lagoon and Lagoon about Drupal. This happens with copying specific YAML and PHP Files into your Git Repository.
+In order for Drupal to work with Lagoon we need to teach Drupal about Lagoon and Lagoon about Drupal. This happens by copying specific YAML and PHP Files into your Git repository.
 
-You find these Files [here](https://github.com/amazeeio/lagoon/tree/master/docs/using_lagoon/drupal). The easiest way is to download them as [ZIP File](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/amazeeio/lagoon/tree/master/docs/using_lagoon/drupal) and copy them into your Git Repository. For each Drupal Version and Database Type you will find an individual folder. A short overview of what they are:
+You find [these Files in our GitHub repository](https://github.com/amazeeio/lagoon/tree/master/docs/using_lagoon/drupal); the easiest way is to [download these files as a ZIP file](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/amazeeio/lagoon/tree/master/docs/using_lagoon/drupal) and copy them into your Git repository. For each Drupal version and database type you will find an individual folder. A short overview of what they are:
 
 - `.lagoon.yml` - The main file that will be used by Lagoon to understand what should be deployed and many more things. This file has some sensible Drupal defaults, if you would like to edit or modify, please check the specific [Documentation for .lagoon.yml](/using_lagoon/lagoon_yml.md)
 - `docker-compose.yml`, `.dockerignore`  and `Dockerfile.*` - These files are used to run your Local Drupal Development environment, they tell docker which services to start and how to build them. They contain sensible defaults and many commented lines, it should be pretty much self describing. If you would like to find out more, see [Documentation for docker-compose.yml]()

docs/using_lagoon/lagoon_yml.md

@@ -85,11 +85,11 @@ This setting allows you to enable injecting the deployed git SHA into your proje
 
 There are different type of tasks you can define, they differ when exactly they are executed in a build flow:
 
-### `pre_rollout.[i].run`
+### Pre-Rollout Tasks - `pre_rollout.[i].run`
 The task defined as `pre_rollout` tasks will run against your project _after_ the new images have been built successfully and _before_ the project gets altered in any way.
 This feature enables you for example to create a database dump before the rollout is running. This will make it easier to roll-back in case of an issue with the rollout.
 
-#### `post_rollout.[i].run`
+#### Post-Rollout Tasks - `post_rollout.[i].run`
 Here you can specify tasks which need to run against your project, _after_:
 
 - all Images have been successfully built
@@ -119,14 +119,20 @@ The simplest route is the `example.com` example above. This will assume that you
 
 In the `"www.example.com"` example, we see two more options (also see the `:` at the end of the route and that the route is wrapped in `"`, that's important!):
 
+#### SSL Configuration - `tls-acme`
+
 * `tls-acme: 'true'` tells Lagoon to issue a Let's Encrypt certificate for that route, this is the default. If you don't like a Let's Encrypt set this to `tls-acme: 'false'`
 * `insecure` can be set to `None`, `Allow` or `Redirect`.
     * `Allow` simply sets up both routes for http and https (this is the default).
     * `Redirect` will redirect any http requests to https
     * `None` will mean a route for http will _not_ be created, and no redirect will take place
 * `hsts` can be set to a value of `max-age=31536000;includeSubDomains;preload`. Ensure there are no spaces and no other parameters included. Only `max-age` parameter is required. The required `max-age` parameter indicates the length of time, in seconds, the HSTS policy is in effect for.
 
-#### `environments.[name].cronjobs`
+!!! Hint
+    If you plan to switch from a SSL certificate signed by a Certificate Authority (CA) to a Let's Encrypt certificate best get in touch with your Lagoon administrator to oversee the transition as there are [known issues](https://github.com/tnozicka/openshift-acme/issues/68) during the transition. Workaround would be manually removing the CA certificate and then trigger the Let's Encrypt process.
+
+
+#### Cronjobs - `environments.[name].cronjobs`
 As most of the time it is not desirable to run the same cronjobs across all environments, you must explicitly define which jobs you want to run for each environment.
 
 * `name:`

images/elasticsearch/Dockerfile

@@ -39,13 +39,25 @@ RUN echo $'xpack.security.enabled: false\n\
 \n\
 node.name: "${HOSTNAME}"\n\
 node.master: "${NODE_MASTER}"\n\
+node.data: "${NODE_DATA}"\n\
+node.ingest: "${NODE_INGEST}"\n\
+node.ml: "${NODE_ML}"\n\
+xpack.ml.enabled: "${XPACK_ML_ENABLED}"\n\
+cluster.remote.connect: "${CLUSTER_REMOTE_CONNECT}"\n\
 discovery.zen.minimum_master_nodes: "${DISCOVERY_ZEN_MINIMUM_MASTER_NODES}"' >> config/elasticsearch.yml
 
 RUN fix-permissions config
 
 ENV ES_JAVA_OPTS="-Xms200m -Xmx200m" \
     DISCOVERY_ZEN_MINIMUM_MASTER_NODES=1 \
-    NODE_MASTER=true
+    NODE_MASTER=true \
+    NODE_DATA=true \
+    NODE_INGEST=true \
+    NODE_ML=true \
+    XPACK_ML_ENABLED=true \
+    CLUSTER_REMOTE_CONNECT=true \
+    EXTRA_OPTS=""
+
 
 VOLUME [ "/usr/share/elasticsearch/data" ]
 

images/elasticsearch/docker-entrypoint.sh

@@ -2,6 +2,10 @@
 
 set -eo pipefail
 
+if [ ! -z "$EXTRA_OPTS" ]; then
+  echo -e "${EXTRA_OPTS}" >> /usr/share/elasticsearch/config/elasticsearch.yml
+fi
+
 if [ -z "$POD_NAMESPACE" ]; then
   # Single container runs in docker
   echo "POD_NAMESPACE not set, spin up single node"
@@ -13,4 +17,4 @@ else
   echo "Using service name: ${K8S_SVC_NAME}"
   # copy the pristine version to the one that can be edited
   /usr/bin/peer-finder -on-start="/lagoon/configure-es.sh" -service=${K8S_SVC_NAME}
-fi
+fi

images/mariadb-galera/Dockerfile

@@ -81,7 +81,7 @@ RUN apk add --no-cache --virtual .common-run-deps \
         zlib-dev && \
     cd /tmp && git clone https://github.com/libcheck/check.git && cd check && mkdir build && cd build && cmake ../ && make && make install && \
     cd /tmp && git clone -b mariadb-3.x https://github.com/MariaDB/galera.git && \
-    cd /tmp/galera && git submodule update --init --jobs=6 && sed -i s/PAGE_SIZE/PAGE_SIZE_64K/g galerautils/src/gu_alloc.cpp && sed -i '/#include <limits>/a #include <stdint.h>' galerautils/src/gu_datetime.hpp && \
+    cd /tmp/galera && git checkout 227e96e457acb60037450bc1e81c45594782e906 && git submodule update --init --jobs=6 && sed -i s/PAGE_SIZE/PAGE_SIZE_64K/g galerautils/src/gu_alloc.cpp && sed -i '/#include <limits>/a #include <stdint.h>' galerautils/src/gu_datetime.hpp && \
     cd /tmp/galera && ./scripts/build.sh --so strict_build_flags=0 && \
     mkdir -p /usr/lib64/galera && mv /tmp/galera/libgalera_smm.so /usr/lib64/galera/libgalera_smm.so && \
     apk del --purge .galera-build-deps && \

images/oc-build-deploy-dind/build-deploy-docker-compose.sh

@@ -460,7 +460,7 @@ else
 fi
 
 # If restic backups are supported by this cluster we create the schedule definition
-if oc auth --insecure-skip-tls-verify can-i create schedules.backup.appuio.ch -q > /dev/null; then
+if oc auth --insecure-skip-tls-verify -n ${OPENSHIFT_PROJECT} can-i create schedules.backup.appuio.ch -q > /dev/null; then
 
   if ! oc --insecure-skip-tls-verify -n ${OPENSHIFT_PROJECT} get secret baas-repo-pw &> /dev/null; then
     # Create baas-repo-pw secret based on the project secret
@@ -750,7 +750,7 @@ do
   fi
 
   # Generate Backup Definitions are supported and if service type defines one
-  if oc auth --insecure-skip-tls-verify can-i create prebackuppod.backup.appuio.ch -q > /dev/null; then
+  if oc auth --insecure-skip-tls-verify -n ${OPENSHIFT_PROJECT} can-i create prebackuppod.backup.appuio.ch -q > /dev/null; then
     OPENSHIFT_SERVICES_TEMPLATE="/oc-build-deploy/openshift-templates/${SERVICE_TYPE}/prebackuppod.yml"
     if [ -f $OPENSHIFT_SERVICES_TEMPLATE ]; then
       OPENSHIFT_TEMPLATE=$OPENSHIFT_SERVICES_TEMPLATE

images/oc-build-deploy-dind/build-deploy.sh

@@ -45,6 +45,8 @@ DEPLOYER_TOKEN=$(cat /var/run/secrets/lagoon/deployer/token)
 oc login --insecure-skip-tls-verify --token="${DEPLOYER_TOKEN}" https://kubernetes.default.svc
 set -x
 
+oc project --insecure-skip-tls-verify $OPENSHIFT_PROJECT
+
 ADDITIONAL_YAMLS=($(cat .lagoon.yml | shyaml keys additional-yaml || echo ""))
 
 for ADDITIONAL_YAML in "${ADDITIONAL_YAMLS[@]}"

images/oc-build-deploy-dind/openshift-templates/cli-persistent/custom-cronjob.yml

@@ -94,6 +94,9 @@ objects:
                 key: appuio.ch/autoscaled
                 operator: Equal
                 value: 'true'
+              - effect: NoSchedule
+                key: lagoon/build
+                operator: Exists
             volumes:
               - name: ${PERSISTENT_STORAGE_NAME}
                 persistentVolumeClaim:

images/oc-build-deploy-dind/openshift-templates/cli-persistent/deployment.yml

@@ -123,6 +123,8 @@ objects:
                 requests:
                   cpu: 10m
                   memory: 10Mi
+                limits:
+                  memory: 8Gi
       test: false
       triggers:
         - type: ConfigChange

images/oc-build-deploy-dind/openshift-templates/cli/custom-cronjob.yml

@@ -83,6 +83,9 @@ objects:
                 key: appuio.ch/autoscaled
                 operator: Equal
                 value: 'true'
+              - effect: NoSchedule
+                key: lagoon/build
+                operator: Exists
             volumes:
               - name: lagoon-sshkey
                 secret:

images/oc-build-deploy-dind/openshift-templates/cli/deployment.yml

@@ -105,6 +105,8 @@ objects:
                 requests:
                   cpu: 10m
                   memory: 10Mi
+                limits:
+                  memory: 8Gi
       test: false
       triggers:
         - type: ConfigChange

images/oc-build-deploy-dind/openshift-templates/node-persistent/custom-cronjob.yml

@@ -91,6 +91,9 @@ objects:
                 key: appuio.ch/autoscaled
                 operator: Equal
                 value: 'true'
+              - effect: NoSchedule
+                key: lagoon/build
+                operator: Exists
             volumes:
               - name: ${SERVICE_NAME}
                 persistentVolumeClaim:

images/oc-build-deploy-dind/openshift-templates/node/custom-cronjob.yml

@@ -82,6 +82,9 @@ objects:
                 key: appuio.ch/autoscaled
                 operator: Equal
                 value: 'true'
+              - effect: NoSchedule
+                key: lagoon/build
+                operator: Exists
             containers:
               - image: ${SERVICE_IMAGE}
                 command:

images/php/cli-drupal/Dockerfile

@@ -7,9 +7,9 @@ LABEL maintainer="amazee.io"
 ENV LAGOON=cli-drupal
 
 # Defining Versions - https://github.com/hechoendrupal/drupal-console-launcher/releases
-ENV DRUPAL_CONSOLE_LAUNCHER_VERSION=1.8.0 \
-    DRUPAL_CONSOLE_LAUNCHER_SHA=db43525189999d2056d4d8bcefaf9600d91df570 \
-    DRUSH_VERSION=8.2.3
+ENV DRUPAL_CONSOLE_LAUNCHER_VERSION=1.9.1 \
+    DRUPAL_CONSOLE_LAUNCHER_SHA=c44be5772de751a498374b43290c693e6a8c79f4\
+    DRUSH_VERSION=8.3.0
 
 RUN curl -L -o /usr/local/bin/drupal "https://github.com/hechoendrupal/drupal-console-launcher/releases/download/${DRUPAL_CONSOLE_LAUNCHER_VERSION}/drupal.phar" \
     && echo "${DRUPAL_CONSOLE_LAUNCHER_SHA} /usr/local/bin/drupal" | sha1sum \

images/php/cli/61-php-xdebug-cli-env.sh

@@ -1,7 +1,7 @@
 #!/bin/sh
 
-# Only if XDEBUG_ENABLE is set
-if [ ${XDEBUG_ENABLE+x} ]; then
+# Only if XDEBUG_ENABLE is not empty
+if [ ! -z ${XDEBUG_ENABLE} ]; then
   # XDEBUG_CONFIG is used by xdebug to decide if an xdebug session should be started in the CLI or not.
   # The content doesn't really matter it just needs to be set, the actual connection details are loaded from /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
   export XDEBUG_CONFIG="idekey=lagoon"

images/php/cli/Dockerfile

@@ -7,8 +7,8 @@ LABEL maintainer="amazee.io"
 ENV LAGOON=cli
 
 # Defining Versions - https://getcomposer.org/download/
-ENV COMPOSER_VERSION=1.8.6 \
-  COMPOSER_HASH_SHA256=b66f9b53db72c5117408defe8a1e00515fe749e97ce1b0ae8bdaa6a5a43dd542
+ENV COMPOSER_VERSION=1.9.0 \
+  COMPOSER_HASH_SHA256=c9dff69d092bdec14dee64df6677e7430163509798895fbd54891c166c5c0875
 
 RUN apk add --no-cache git \
         unzip \
@@ -20,10 +20,10 @@ RUN apk add --no-cache git \
         patch \
         procps \
         coreutils \
+        mariadb-client \
         postgresql-client \
         openssh-sftp-server \
         findutils \
-    && apk add --no-cache "mariadb-client=10.2.24-r0" --repository http://dl-cdn.alpinelinux.org/alpine/v3.8/main/ \
     && apk add --no-cache nodejs-current nodejs-npm yarn --force-overwrite --repository http://dl-cdn.alpinelinux.org/alpine/edge/main/ --repository http://dl-cdn.alpinelinux.org/alpine/edge/community/ \
     && ln -s /usr/lib/ssh/sftp-server /usr/local/bin/sftp-server \
     && rm -rf /var/cache/apk/* \

images/php/fpm/entrypoints/60-php-xdebug.sh

@@ -25,12 +25,12 @@ get_dockerhost() {
   return
 }
 
-# Only if XDEBUG_ENABLE is set
-if [ ${XDEBUG_ENABLE+x} ]; then
+# Only if XDEBUG_ENABLE is not empty
+if [ ! -z ${XDEBUG_ENABLE} ]; then
   # remove first line and all comments
   sed -i '1d; s/;//' /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
   # add comment that explains how we have xdebug enabled
-  sed -i '1s/^/;xdebug enabled as XDEBUG_ENABLE is set, see \/lagoon\/entrypoints\/60-php-xdebug.sh \n/' /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
+  sed -i '1s/^/;xdebug enabled as XDEBUG_ENABLE is not empty, see \/lagoon\/entrypoints\/60-php-xdebug.sh \n/' /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
 
   # Only if DOCKERHOST is not already set, allows to set a DOCKERHOST via environment variables
   if [[ -z ${DOCKERHOST+x} ]]; then

lagoon-remote/docker-compose.yaml

@@ -25,7 +25,7 @@ services:
       lagoon.rollout: daemonset
       lagoon.template: logs-collector/.lagoon.yml
   docker-host:
-    image: amazeeiolagoon/master-docker-host
+    image: amazeeio/docker-host:latest
     labels:
       lagoon.type: custom
       lagoon.template: docker-host/docker-host.yaml

node-packages/commons/package.json

@@ -26,6 +26,7 @@
     "jsonwebtoken": "^8.0.1",
     "kubernetes-client": "^3.15.0",
     "lokka": "^1.7.0",
+    "node-fetch": "^2.6.0",
     "ramda": "^0.25.0",
     "winston": "^2.4.0",
     "winston-logstash": "^0.4.0"

node-packages/commons/src/api.js

@@ -11,7 +11,7 @@ import type {
 } from './types';
 
 const { Lokka } = require('lokka');
-const { Transport } = require('@lagoon/lokka-transport-http');
+const { Transport } = require('./lokka-transport-http-retry');
 const R = require('ramda');
 const { createJWTWithoutUserId } = require('./jwt');
 const { logger } = require('./local-logging');
@@ -249,6 +249,25 @@ const getAllEnvironmentBackups = (): Promise<Project[]> =>
 `,
   );
 
+const getEnvironmentBackups = (openshiftProjectName: string): Promise<Project[]> =>
+  graphqlapi.query(
+    `
+  query environmentByOpenshiftProjectName($openshiftProjectName: String!) {
+    environmentByOpenshiftProjectName(openshiftProjectName: $openshiftProjectName) {
+      id
+      name
+      openshiftProjectName
+      project {
+        name
+      }
+      backups {
+        ...${backupFragment}
+      }
+    }
+  }
+`, { openshiftProjectName }
+  );
+
 const updateCustomer = (id: number, patch: CustomerPatch): Promise<Object> =>
   graphqlapi.mutate(
     `
@@ -965,6 +984,7 @@ module.exports = {
   deleteBackup,
   updateRestore,
   getAllEnvironmentBackups,
+  getEnvironmentBackups,
   updateUser,
   deleteUser,
   addUserToCustomer,

node-packages/commons/src/lokka-transport-http-retry.js

@@ -0,0 +1,58 @@
+const {
+  Transport: LokkaTransportHttp,
+} = require('@lagoon/lokka-transport-http');
+const fetchUrl = require('node-fetch');
+
+class NetworkError extends Error {}
+class ApiError extends Error {}
+
+// Retries the fetch if operational/network errors occur
+const retryFetch = (endpoint, options, retriesLeft = 5, interval = 1000) =>
+  new Promise((resolve, reject) =>
+    fetchUrl(endpoint, options)
+      .then(response => {
+        if (response.status !== 200 && response.status !== 400) {
+          throw new NetworkError(`Invalid status code: ${response.status}`);
+        }
+
+        return response.json();
+      })
+      .then(({ data, errors }) => {
+        if (errors) {
+          const error = new ApiError(`GraphQL Error: ${errors[0].message}`);
+          error.rawError = errors;
+          error.rawData = data;
+          throw error;
+        }
+
+        resolve(data);
+      })
+      .catch(error => {
+        // Don't retry if limit is reached or the error was not network/operational
+        if (retriesLeft === 1 || error instanceof ApiError) {
+          reject(error);
+          return;
+        }
+
+        setTimeout(() => {
+          retryFetch(endpoint, options, retriesLeft - 1).then(resolve, reject);
+        }, interval);
+      }),
+  );
+
+class Transport extends LokkaTransportHttp {
+  constructor(endpoint, options = {}) {
+    super(endpoint, options);
+  }
+
+  send(query, variables, operationName) {
+    const payload = { query, variables, operationName };
+    const options = this._buildOptions(payload);
+
+    return retryFetch(this.endpoint, options);
+  }
+}
+
+module.exports = {
+  Transport,
+};

services/docker-host/docker-host.yaml

@@ -39,7 +39,7 @@ parameters:
     value: "Recreate"
   - name: SERVICE_IMAGE
     description: Pullable image of service
-    value: amazeeiolagoon/master-docker-host
+    value: amazeeio/docker-host:latest
 objects:
 - apiVersion: v1
   kind: DeploymentConfig
@@ -83,7 +83,7 @@ objects:
             - name: CRONJOBS
               value: |
                 22 1 * * * /lagoon/cronjob.sh "/prune-images.sh"
-                22 0 * * * /lagoon/cronjob.sh "/remove-exited.sh"
+                22 */4 * * * /lagoon/cronjob.sh "/remove-exited.sh"
                 */15 * * * * /lagoon/cronjob.sh "/update-push-images.sh"
           ports:
           - containerPort: 2375

services/logs-forwarder/.lagoon.multi.yml

@@ -252,6 +252,8 @@ objects:
             total_limit_size 15GB
             flush_thread_count 8
             overflow_action block
+            retry_type periodic
+            retry_wait 10s
           </buffer>
           id_key viaq_msg_id
           remove_keys viaq_msg_id