Merge pull request #35918 from iocanel/wait-for-config-and-doc

Document init tasks

docs/src/main/asciidoc/flyway.adoc

@@ -285,6 +285,7 @@ You can find more information about this feature in the xref:hibernate-orm.adoc#
 
 == Flyway on Kubernetes
 Sometimes, it's helpful not to execute Flyway initialization on each application startup. One such example is when deploying
+
 on Kubernetes, where it doesn't make sense to execute Flyway on every single replica. Instead it's desirable to execute it
 once and then start the actual application without Flyway. To support this use case, when generating manifests for Kubernetes
 the generated manifests contain a Kubernetes initialization `Job` for Flyway.
@@ -312,15 +313,15 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
 
 [source,properties]
 ----
-quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
 or on Openshift:
 
 
 [source,properties]
 ----
-quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
 **Note**: In this context globally means `for all extensions that support init task externalization`.

docs/src/main/asciidoc/init-tasks.adoc

@@ -0,0 +1,145 @@
+////
+This guide is maintained in the main Quarkus repository
+and pull requests should be submitted there:
+https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
+////
+= Initialization tasks
+:categories: initialization
+:summary: This reference guide explains how to configure initialization tasks
+
+There are often initialization tasks performed by Quarkus extensions that are meant to be run once.
+For example, Flyway or Liquibase initialization falls into that category. But what happens when the scaling
+needs of an application requires more instances of the application to run? Or what happens when the application
+restarts ?
+
+A common environment where both of these cases are pretty common is Kubernetes. To address these challenges,
+Quarkus allows externalization of such tasks as Kubernetes https://kubernetes.io/docs/concepts/workloads/controllers/job/[Jobs] and uses https://kubernetes.io/docs/concepts/workloads/pods/init-containers/[init containers] to ensure that an 
+application instance only starts once the initialization jobs have finished. With this approach even if an 
+application has multiple replicas, the initialization logic will only run once.
+
+This approach is reflected in the manifests generated by xref:kubernetes.adoc[Kubernetes extension].
+
+== Disabling the feature
+
+The feature can be explictily disabled per task (enabled by default).
+The default behavior can change by setting the following property to `false`:
+
+[source,properties]
+----
+quarkus.kubernetes.init-task-defaults.enabled=false
+----
+
+or on Openshift:
+
+[source,properties]
+----
+quarkus.openshift.init-task-defaults.enabled=false
+----
+
+**Note**: All the configuration options in this guide are available on both OpenShift and Kubernetes. The rest of the guide will use Kubernetes(`quarkus.kubernetes` prefix) 
+configuration prefix, but all the configuration options are also available for OpenShift(`quarkus.openshift` prefix) too.
+
+In the case where we need to disable a particular task, we can use the following property:
+
+[source,properties]
+----
+quarkus.kubernetes.init-tasks."<task name>".enabled=false
+----
+
+The task name is the name of the extension that performs the initialization.
+Examples:
+
+For Flyway:
+
+[source,properties]
+----
+quarkus.kubernetes.init-tasks.flyway.enabled=false
+----
+
+For Liquibase:
+
+[source,properties]
+----
+quarkus.kubernets.init-tasks.liquibase.enabled=false
+----
+
+For Liquibase Mongodb:
+
+[source,properties]
+----
+quarkus.kubernetes.init-tasks.liquibase-mongodb.enabled=false
+----
+
+
+== Controlling the generated job
+
+The job container is pretty similar to the application container, and the only thing that changes is the configured environment variables.
+More specifically, the following environment variable is added, to tell the job to exit right after initialization.
+
+[source,properties]
+----
+QUARKUS_INIT_AND_EXIT=true
+----
+
+The image, image pull policy, service account, volumes, mounts and additional environment variables are inherited/copied from the deployment resource.
+Any customization to the original deployment resource (via configuration or extension) will also be reflected in the job.
+
+== Controlling the generated init container
+
+The name of the generated init container is `wait-for-${task name}` by default.
+Given that the init container is part of the same pod as the actual application it will get the same service account (and therefore permissions) and volumes as the application.
+Further customization to the container can be done using using the configuration options for init containers (see `quarkus.kubernetes.init-containers` or `quarkus.openshift.init-containers`).
+
+Examples:
+
+To set the imagePullPolicy to `IfNotPresent` on the init container that waits for the `flyway` job:
+
+[source,properties]
+----
+quarkus.kubernetes.init-containers.wait-for-flyway.image-pull-policy=IfNotPresent
+----
+
+To set custom command (say `custom-wait-for`) on the init container that waits for the `flyway` job:
+
+[source,properties]
+----
+quarkus.kubernetes.init-containers.wait-for-flyway.command=custom-wait-for
+----
+
+
+== Orchestration of the initialization tasks
+
+The deployment resource should not start until the job has been completed. The typical pattern that is used among Kubernetes users is the
+use of init containers to achieve this. An init container that `wait for` the job to complete is enough to enforce that requirement.
+
+=== Using a custom wait-for container image
+
+To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-root-v1.7` you can use:
+
+[source,properties]
+----
+quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
+----
+
+To change the `wait-for` image for a particular init container (e.g. `wait-for-flway`) you can use:
+
+[source,properties]
+----
+quarkus.kubernetes.init-containers.wait-for-flyway=my/wait-for-image:1.0
+----
+
+=== Configuring permissions
+
+For an init container to be able to perform the `wait for job` it needs to be able to perform `get` operations on the job resource.
+This is done automatically and the generated manifests include the required `Role` and `RoleBinding` resources. 
+
+If for any reason additional permissions are required either by the init container or the job, they can be configured with through the xref:deploying-to-kuberentes.adoc#generating-rbac-resources[Kubernetes RBAC configuration].
+
+**Note**: The application, the init container and the job use the same `ServiceAccount` and therefore, share the same permissions. 
+
+== Extension providing Initialization Tasks
+
+Currently, this feature is used by the following extensions:
+- xref:flyway.adoc[Flyway]
+- xref:liquibase.adoc[Liquibase]
+- xref:liquibase-mongodb.adoc[Liquibase MongoDB]

docs/src/main/asciidoc/liquibase-mongodb.adoc

@@ -181,15 +181,15 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
 
 [source,properties]
 ----
-quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
 or on Openshift:
 
 
 [source,properties]
 ----
-quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
 **Note**: In this context globally means `for all extensions that support init task externalization`.

docs/src/main/asciidoc/liquibase.adoc

@@ -259,19 +259,17 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
 
 [source,properties]
 ----
-quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
 or on Openshift:
 
 
 [source,properties]
 ----
-quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
+quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
 ----
 
-
-
 **Note**: In this context globally means `for all extensions that support init task externalization`.
 
 == Configuration Reference

extensions/kubernetes/kind/deployment/src/main/java/io/quarkus/kind/deployment/KindProcessor.java

@@ -18,6 +18,7 @@
 import io.quarkus.deployment.annotations.BuildProducer;
 import io.quarkus.deployment.annotations.BuildStep;
 import io.quarkus.deployment.builditem.ApplicationInfoBuildItem;
+import io.quarkus.deployment.builditem.InitTaskBuildItem;
 import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
 import io.quarkus.deployment.pkg.PackageConfig;
 import io.quarkus.deployment.pkg.builditem.ArtifactResultBuildItem;
@@ -26,6 +27,7 @@
 import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
 import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
 import io.quarkus.kubernetes.deployment.DevClusterHelper;
+import io.quarkus.kubernetes.deployment.InitTaskProcessor;
 import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
 import io.quarkus.kubernetes.deployment.KubernetesConfig;
 import io.quarkus.kubernetes.deployment.ResourceNameUtil;
@@ -135,4 +137,25 @@ public void postBuild(ContainerImageInfoBuildItem image, List<ContainerImageBuil
         //So, we now always perform this step
         ExecUtil.exec("kind", "load", "docker-image", image.getImage());
     }
-}
+
+    @BuildStep
+    void externalizeInitTasks(
+            ApplicationInfoBuildItem applicationInfo,
+            KubernetesConfig config,
+            ContainerImageInfoBuildItem image,
+            List<InitTaskBuildItem> initTasks,
+            BuildProducer<KubernetesJobBuildItem> jobs,
+            BuildProducer<KubernetesInitContainerBuildItem> initContainers,
+            BuildProducer<KubernetesEnvBuildItem> env,
+            BuildProducer<KubernetesRoleBuildItem> roles,
+            BuildProducer<KubernetesRoleBindingBuildItem> roleBindings,
+            BuildProducer<KubernetesServiceAccountBuildItem> serviceAccount,
+
+            BuildProducer<DecoratorBuildItem> decorators) {
+        final String name = ResourceNameUtil.getResourceName(config, applicationInfo);
+        if (config.isExternalizeInit()) {
+            InitTaskProcessor.process(KIND, name, image, initTasks, config.getInitTaskDefaults(), config.getInitTasks(),
+                    jobs, initContainers, env, roles, roleBindings, serviceAccount, decorators);
+        }
+    }
+}

extensions/kubernetes/minikube/deployment/src/main/java/io/quarkus/minikube/deployment/MinikubeProcessor.java

@@ -17,12 +17,14 @@
 import io.quarkus.deployment.annotations.BuildProducer;
 import io.quarkus.deployment.annotations.BuildStep;
 import io.quarkus.deployment.builditem.ApplicationInfoBuildItem;
+import io.quarkus.deployment.builditem.InitTaskBuildItem;
 import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
 import io.quarkus.deployment.pkg.PackageConfig;
 import io.quarkus.deployment.pkg.builditem.OutputTargetBuildItem;
 import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
 import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
 import io.quarkus.kubernetes.deployment.DevClusterHelper;
+import io.quarkus.kubernetes.deployment.InitTaskProcessor;
 import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
 import io.quarkus.kubernetes.deployment.KubernetesConfig;
 import io.quarkus.kubernetes.deployment.ResourceNameUtil;
@@ -122,4 +124,25 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
                 livenessPath, readinessPath, startupPath,
                 roles, clusterRoles, serviceAccounts, roleBindings, customProjectRoot);
     }
-}
+
+    @BuildStep
+    void externalizeInitTasks(
+            ApplicationInfoBuildItem applicationInfo,
+            KubernetesConfig config,
+            ContainerImageInfoBuildItem image,
+            List<InitTaskBuildItem> initTasks,
+            BuildProducer<KubernetesJobBuildItem> jobs,
+            BuildProducer<KubernetesInitContainerBuildItem> initContainers,
+            BuildProducer<KubernetesEnvBuildItem> env,
+            BuildProducer<KubernetesRoleBuildItem> roles,
+            BuildProducer<KubernetesRoleBindingBuildItem> roleBindings,
+            BuildProducer<KubernetesServiceAccountBuildItem> serviceAccount,
+
+            BuildProducer<DecoratorBuildItem> decorators) {
+        final String name = ResourceNameUtil.getResourceName(config, applicationInfo);
+        if (config.isExternalizeInit()) {
+            InitTaskProcessor.process(MINIKUBE, name, image, initTasks, config.getInitTaskDefaults(), config.getInitTasks(),
+                    jobs, initContainers, env, roles, roleBindings, serviceAccount, decorators);
+        }
+    }
+}

extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/InitTaskConfig.java

@@ -15,15 +15,15 @@ public class InitTaskConfig {
 
     /**
      * The init task image to use by the init-container.
+     * Deprecated, use waitForContainer.image instead.
      */
     @Deprecated
     @ConfigItem
     public Optional<String> image;
 
     /**
-     * The init task image to use by the init-container.
+     * The configuration of the `wait for` container.
      */
-    @ConfigItem(defaultValue = "groundnuty/k8s-wait-for:no-root-v1.7")
-    public String waitForImage;
-
+    @ConfigItem
+    public InitTaskContainerConfig waitForContainer;
 }

extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/InitTaskContainerConfig.java

@@ -0,0 +1,15 @@
+package io.quarkus.kubernetes.deployment;
+
+import io.quarkus.runtime.annotations.ConfigGroup;
+import io.quarkus.runtime.annotations.ConfigItem;
+
+@ConfigGroup
+public class InitTaskContainerConfig {
+
+    /**
+     * The init task image to use by the init-container.
+     */
+    @ConfigItem(defaultValue = "groundnuty/k8s-wait-for:no-root-v1.7")
+    public String image;
+
+}

extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/InitTaskProcessor.java

@@ -3,6 +3,7 @@
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
+import java.util.regex.Pattern;
 
 import io.dekorate.kubernetes.config.EnvBuilder;
 import io.dekorate.kubernetes.decorator.AddEnvVarDecorator;
@@ -21,9 +22,9 @@
 
 public class InitTaskProcessor {
 
-    private static final String INIT_CONTAINER_WAITER_NAME = "init";
+    private static final String INIT_CONTAINER_WAITER_NAME = "wait-for-";
 
-    static void process(
+    public static void process(
             String target, // kubernetes, openshift, etc.
             String name,
             ContainerImageInfoBuildItem image,
@@ -40,7 +41,12 @@ static void process(
 
         boolean generateRoleForJobs = false;
         for (InitTaskBuildItem task : initTasks) {
-            InitTaskConfig config = initTasksConfig.getOrDefault(task.getName(), initTaskDefaults);
+            String taskName = task.getName()
+                    //Strip appplication.name prefix and init suffix (for compatibility with previous versions)
+                    .replaceAll("^" + Pattern.quote(name + "-"), "")
+                    .replaceAll(Pattern.quote("-init") + "$", "");
+            String jobName = name + "-" + taskName + "-init";
+            InitTaskConfig config = initTasksConfig.getOrDefault(taskName, initTaskDefaults);
             if (config == null || config.enabled) {
                 generateRoleForJobs = true;
                 jobs.produce(KubernetesJobBuildItem.create(image.getImage())
@@ -60,10 +66,11 @@ static void process(
                                     .build())));
                 });
 
-                String waitForImage = config.image.orElse(config.waitForImage);
-                initContainers.produce(KubernetesInitContainerBuildItem.create(INIT_CONTAINER_WAITER_NAME, waitForImage)
-                        .withTarget(target)
-                        .withArguments(List.of("job", task.getName())));
+                String waitForImage = config.image.orElse(config.waitForContainer.image);
+                initContainers
+                        .produce(KubernetesInitContainerBuildItem.create(INIT_CONTAINER_WAITER_NAME + taskName, waitForImage)
+                                .withTarget(target)
+                                .withArguments(List.of("job", jobName)));
             }
         }