Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the generation init-tasks by database migration on Kubernetes #33116

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -221,13 +221,15 @@ public ServiceStartBuildItem startActions(FlywayRecorder recorder,
}

@BuildStep
public InitTaskBuildItem configureInitTask() {
return InitTaskBuildItem.create()
.withName("flyway-init")
.withTaskEnvVars(Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_FLYWAY_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_FLYWAY_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true);
public void configureInitTask(FlywayBuildTimeConfig config, BuildProducer<InitTaskBuildItem> initTasks) {
if (config.generateInitTask && config.defaultDataSource.migrateWithInitTask) {
initTasks.produce(InitTaskBuildItem.create()
.withName("flyway-init")
.withTaskEnvVars(Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_FLYWAY_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_FLYWAY_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true));
}
}

private Set<String> getDataSourceNames(List<JdbcDataSourceBuildItem> jdbcDataSourceBuildItems) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,15 @@ public FlywayDataSourceBuildTimeConfig getConfigForDataSourceName(String dataSou
return namedDataSources.getOrDefault(dataSourceName, FlywayDataSourceBuildTimeConfig.defaultConfig());
}

/**
* Flag to enable / disable the generation of the init task Kubernetes resources.
* This property is only relevant if the Quarkus Kubernetes/OpenShift extensions are present.
*
* The default value is `quarkus.flyway.enabled`.
*/
@ConfigItem(defaultValue = "${quarkus.flyway.enabled:true}")
public boolean generateInitTask;

/**
* Flyway configuration for the default datasource.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,15 @@ public final class FlywayDataSourceBuildTimeConfig {
@ConfigItem
public Optional<List<String>> callbacks = Optional.empty();

/**
* Flag to enable / disable the migration using the generated init task Kubernetes resources.
* This property is only relevant if the Quarkus Kubernetes/OpenShift extensions are present.
*
* The default value is `quarkus.flyway.migrate-at-start`.
*/
@ConfigItem(defaultValue = "${quarkus.flyway.migrate-at-start:false}")
public boolean migrateWithInitTask;

/**
* Creates a {@link FlywayDataSourceBuildTimeConfig} with default settings.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ public final class KubernetesInitContainerBuildItem extends MultiBuildItem {
private final boolean sharedEnvironment;
private final boolean sharedFilesystem;

public static KubernetesInitContainerBuildItem create(String image) {
return new KubernetesInitContainerBuildItem("init", null, image, Collections.emptyList(), Collections.emptyList(),
public static KubernetesInitContainerBuildItem create(String name, String image) {
return new KubernetesInitContainerBuildItem(name, null, image, Collections.emptyList(), Collections.emptyList(),
Collections.emptyMap(), false, false);
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
package io.quarkus.kubernetes.deployment;

import io.dekorate.kubernetes.adapter.ContainerAdapter;
import io.dekorate.kubernetes.config.Container;
import io.dekorate.kubernetes.decorator.NamedResourceDecorator;
import io.fabric8.kubernetes.api.model.ContainerBuilder;
import io.fabric8.kubernetes.api.model.ObjectMeta;
import io.fabric8.kubernetes.api.model.PodSpecBuilder;

public abstract class AddInitContainerDecorator extends NamedResourceDecorator<PodSpecBuilder> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason for creating a new decorator, let alone 3.
If there is a technical reason why we do so, it should definitely need to be part of a comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary to overwrite the hardcoded properties (image, command, args) of the init-containers using the standard properties: #33097 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need new decorators to do that. Existing decorators for setting image, command and args do already exist and should be used instead. We should limit the amount of overlapping decorators as its complicates predicting the order of things.


private final Container container;

public AddInitContainerDecorator(String deployment, Container container) {
super(deployment);
this.container = container;
}

@Override
public void andThenVisit(PodSpecBuilder podSpec, ObjectMeta resourceMeta) {
var resource = ContainerAdapter.adapt(container);
if (podSpec.hasMatchingInitContainer(this::hasInitContainer)) {
update(podSpec, resource);
} else {
add(podSpec, resource);
}
}

private void add(PodSpecBuilder podSpec, io.fabric8.kubernetes.api.model.Container resource) {
podSpec.addToInitContainers(resource);
}

private void update(PodSpecBuilder podSpec, io.fabric8.kubernetes.api.model.Container resource) {
var matching = podSpec.editMatchingInitContainer(this::hasInitContainer);
if (resource.getImage() != null) {
matching.withImage(resource.getImage());
}

if (resource.getImage() != null) {
matching.withImage(resource.getImage());
}

if (resource.getWorkingDir() != null) {
matching.withWorkingDir(resource.getWorkingDir());
}

if (resource.getCommand() != null && !resource.getCommand().isEmpty()) {
matching.withCommand(resource.getCommand());
}

if (resource.getArgs() != null && !resource.getArgs().isEmpty()) {
matching.withArgs(resource.getArgs());
}

if (resource.getReadinessProbe() != null) {
matching.withReadinessProbe(resource.getReadinessProbe());
}

if (resource.getLivenessProbe() != null) {
matching.withLivenessProbe(resource.getLivenessProbe());
}

matching.addAllToEnv(resource.getEnv());
if (resource.getPorts() != null && !resource.getPorts().isEmpty()) {
matching.withPorts(resource.getPorts());
}

matching.endInitContainer();
}

private boolean hasInitContainer(ContainerBuilder containerBuilder) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This predicate does not test the existence of a container, but wether a container has matching name. The name should probably change to match the functionality.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will rename it

return containerBuilder.getName().equals(container.getName());
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package io.quarkus.kubernetes.deployment;

import io.dekorate.kubernetes.config.Container;
import io.dekorate.kubernetes.decorator.Decorator;

public class AddInitContainerFromExtensionsDecorator extends AddInitContainerDecorator {

public AddInitContainerFromExtensionsDecorator(String deployment, Container container) {
super(deployment, container);
}

public Class<? extends Decorator>[] before() {
return new Class[] { AddInitContainerFromUserConfigDecorator.class };
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
package io.quarkus.kubernetes.deployment;

import io.dekorate.kubernetes.config.Container;

public class AddInitContainerFromUserConfigDecorator extends AddInitContainerDecorator {

public AddInitContainerFromUserConfigDecorator(String name, Container container) {
super(name, container);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@

import io.dekorate.ConfigReference;
import io.dekorate.WithConfigReferences;
import io.dekorate.kubernetes.decorator.AddInitContainerDecorator;
import io.dekorate.kubernetes.decorator.AddSidecarDecorator;
import io.dekorate.kubernetes.decorator.ApplicationContainerDecorator;
import io.dekorate.kubernetes.decorator.ApplyImageDecorator;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package io.quarkus.kubernetes.deployment;

import io.dekorate.kubernetes.decorator.AddEnvVarDecorator;
import io.dekorate.kubernetes.decorator.AddInitContainerDecorator;
import io.dekorate.kubernetes.decorator.AddLivenessProbeDecorator;
import io.dekorate.kubernetes.decorator.AddMountDecorator;
import io.dekorate.kubernetes.decorator.AddPortDecorator;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,23 @@
package io.quarkus.kubernetes.deployment;

import io.dekorate.kubernetes.decorator.*;
import io.dekorate.kubernetes.decorator.AddAwsElasticBlockStoreVolumeDecorator;
import io.dekorate.kubernetes.decorator.AddAzureDiskVolumeDecorator;
import io.dekorate.kubernetes.decorator.AddEnvVarDecorator;
import io.dekorate.kubernetes.decorator.AddLivenessProbeDecorator;
import io.dekorate.kubernetes.decorator.AddMountDecorator;
import io.dekorate.kubernetes.decorator.AddPortDecorator;
import io.dekorate.kubernetes.decorator.AddPvcVolumeDecorator;
import io.dekorate.kubernetes.decorator.AddReadinessProbeDecorator;
import io.dekorate.kubernetes.decorator.AddSidecarDecorator;
import io.dekorate.kubernetes.decorator.ApplyApplicationContainerDecorator;
import io.dekorate.kubernetes.decorator.ApplyArgsDecorator;
import io.dekorate.kubernetes.decorator.ApplyCommandDecorator;
import io.dekorate.kubernetes.decorator.ApplyImageDecorator;
import io.dekorate.kubernetes.decorator.ApplyImagePullPolicyDecorator;
import io.dekorate.kubernetes.decorator.ApplyServiceAccountNamedDecorator;
import io.dekorate.kubernetes.decorator.ApplyWorkingDirDecorator;
import io.dekorate.kubernetes.decorator.Decorator;
import io.dekorate.kubernetes.decorator.NamedResourceDecorator;
import io.dekorate.openshift.decorator.ApplyDeploymentTriggerDecorator;
import io.fabric8.kubernetes.api.model.ObjectMeta;
import io.fabric8.openshift.api.model.DeploymentConfigSpecFluent;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package io.quarkus.kubernetes.deployment;

import java.util.Arrays;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

Expand All @@ -20,22 +20,26 @@

public class InitTaskProcessor {

private static final String INIT_CONTAINER_WAITER_NAME = "init";
private static final String INIT_CONTAINER_WAITER_DEFAULT_IMAGE = "groundnuty/k8s-wait-for:no-root-v1.7";

static void process(
String target, // kubernetes, openshift, etc.
String name,
ContainerImageInfoBuildItem image, List<InitTaskBuildItem> initTasks,
ContainerImageInfoBuildItem image,
List<InitTaskBuildItem> initTasks,
BuildProducer<KubernetesJobBuildItem> jobs,
BuildProducer<KubernetesInitContainerBuildItem> initContainers,
BuildProducer<KubernetesEnvBuildItem> env,
BuildProducer<KubernetesRoleBuildItem> roles,
BuildProducer<KubernetesRoleBindingBuildItem> roleBindings,
BuildProducer<DecoratorBuildItem> decorators) {

initTasks.forEach(task -> {
initContainers.produce(KubernetesInitContainerBuildItem.create("groundnuty/k8s-wait-for:1.3")
.withTarget(target)
.withArguments(Arrays.asList("job", task.getName())));
List<String> initContainerWaiterArgs = new ArrayList<>(initTasks.size() + 1);
initContainerWaiterArgs.add("job");

initTasks.forEach(task -> {
initContainerWaiterArgs.add(task.getName());
jobs.produce(KubernetesJobBuildItem.create(image.getImage())
.withName(task.getName())
.withTarget(target)
Expand Down Expand Up @@ -63,5 +67,12 @@ static void process(
roleBindings.produce(new KubernetesRoleBindingBuildItem(null, "view-jobs", false, target));

});

if (!initTasks.isEmpty()) {
initContainers.produce(KubernetesInitContainerBuildItem.create(INIT_CONTAINER_WAITER_NAME,
INIT_CONTAINER_WAITER_DEFAULT_IMAGE)
.withTarget(target)
.withArguments(initContainerWaiterArgs));
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@
import io.dekorate.kubernetes.decorator.AddEnvVarDecorator;
import io.dekorate.kubernetes.decorator.AddHostAliasesDecorator;
import io.dekorate.kubernetes.decorator.AddImagePullSecretDecorator;
import io.dekorate.kubernetes.decorator.AddInitContainerDecorator;
import io.dekorate.kubernetes.decorator.AddLabelDecorator;
import io.dekorate.kubernetes.decorator.AddLivenessProbeDecorator;
import io.dekorate.kubernetes.decorator.AddMetadataToTemplateDecorator;
Expand Down Expand Up @@ -602,7 +601,7 @@ public void andThenVisit(ContainerBuilder builder) {
}

result.add(new DecoratorBuildItem(target,
new AddInitContainerDecorator(name, containerBuilder
new AddInitContainerFromExtensionsDecorator(name, containerBuilder
.addAllToEnvVars(item.getEnvVars().entrySet().stream().map(e -> new EnvBuilder()
.withName(e.getKey())
.withValue(e.getValue())
Expand Down Expand Up @@ -750,7 +749,8 @@ private static List<DecoratorBuildItem> createPodDecorators(Optional<Project> pr
});

config.getInitContainers().entrySet().forEach(e -> {
result.add(new DecoratorBuildItem(target, new AddInitContainerDecorator(name, ContainerConverter.convert(e))));
result.add(new DecoratorBuildItem(target,
new AddInitContainerFromUserConfigDecorator(name, ContainerConverter.convert(e))));
});

config.getSidecars().entrySet().forEach(e -> {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
import io.dekorate.kubernetes.config.KubernetesConfigBuilder;
import io.dekorate.kubernetes.configurator.ApplyDeployToApplicationConfiguration;
import io.dekorate.kubernetes.decorator.AddIngressDecorator;
import io.dekorate.kubernetes.decorator.AddInitContainerDecorator;
import io.dekorate.kubernetes.decorator.AddServiceResourceDecorator;
import io.dekorate.kubernetes.decorator.ApplyHeadlessDecorator;
import io.dekorate.kubernetes.decorator.ApplyImageDecorator;
Expand Down Expand Up @@ -108,7 +107,7 @@ protected void addDecorators(String group, KubernetesConfig config) {
super.addDecorators(group, config);

for (Container container : config.getInitContainers()) {
resourceRegistry.decorate(group, new AddInitContainerDecorator(config.getName(), container));
resourceRegistry.decorate(group, new AddInitContainerFromUserConfigDecorator(config.getName(), container));
}

if (config.getPorts().length > 0) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -254,14 +254,16 @@ ServiceStartBuildItem startLiquibase(LiquibaseMongodbRecorder recorder,
}

@BuildStep
public InitTaskBuildItem configureInitTask() {
return InitTaskBuildItem.create()
.withName("liquibase-mongodb-init")
.withTaskEnvVars(
Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_LIQUIBASE_MONGODB_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_LIQUIBASE_MONGODB_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true);
public void configureInitTask(LiquibaseMongodbBuildTimeConfig config, BuildProducer<InitTaskBuildItem> initTasks) {
if (config.generateInitTask && config.migrateWithInitTask) {
initTasks.produce(InitTaskBuildItem.create()
.withName("liquibase-mongodb-init")
.withTaskEnvVars(
Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_LIQUIBASE_MONGODB_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_LIQUIBASE_MONGODB_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true));
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,22 @@ public class LiquibaseMongodbBuildTimeConfig {
*/
@ConfigItem(defaultValue = "db/changeLog.xml")
public String changeLog;

/**
* Flag to enable / disable the generation of the init task Kubernetes resources.
* This property is only relevant if the Quarkus Kubernetes/OpenShift extensions are present.
*
* The default value is `quarkus.liquibase-mongodb.enabled`.
*/
@ConfigItem(defaultValue = "${quarkus.liquibase-mongodb.enabled:true}")
public boolean generateInitTask;

/**
* Flag to enable / disable the migration using the generated init task Kubernetes resources.
* This property is only relevant if the Quarkus Kubernetes/OpenShift extensions are present.
*
* The default value is `quarkus.liquibase-mongodb.migrate-at-start`.
*/
@ConfigItem(defaultValue = "${quarkus.liquibase-mongodb.migrate-at-start:false}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need two properties for doing pretty much the same thing.
One should be enough.

Also, I am not a big fan for having build time configuration defaulting to runtime values. This seems weird if not wrong.

Last, I am wondering if generateInitTask will is descriptive enough to let people understand this refers to kubernetes jobs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I am not a big fan for having build time configuration defaulting to runtime values. This seems weird if not wrong.

I'm not super happy with this either. Though, we're generating build time resources (the init-task resources) that is actually disabled by default (migrate-at-start property is disabled by default), and can be enabled/disabled at runtime.

Therefore, if users enable the migrate-at-start property at build time, then it means that they are likely to need the init-task when running the app in K8s, so I think it makes sense to me.

About having two properties, this extension can be enabled/disabled using two properties enabled and migrate-at-start, so having only one property is probably not enough. Also, adding a totally new build-time property will be unlikely to be used by users (because they are used to use the existing and well-known enabled and migrate-at-start properties).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migrate-at-start property is a runtime property and does not affect in any way the generation of kubernetes resources, so the property is redundant and needs to be removed. What the init container does should not be a concern of the part that is generating the Job.

About the default value I don't have a super strong opinion so let's see what others think.
@geoand @gsmet: We need your expert opinion: Is it ok to have a build time property with a default value that points to a runtime property?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure that won't work, but I'll have a look at this tomorrow

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migration is enabled/disabled by runtime properties, and the generated Job to perform the migration has to be created/or not created at build time. So, here is the clash.

Is it ok to have a build time property with a default value that points to a runtime property?

It works as you can see with the new test cases I added as part of this pull request.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migration is enabled/disabled by runtime properties, and the generated Job to perform the migration has to be created/or not created at build time. So, here is the clash.

Agree. There is a clash so we shouldn't tie the two together.

It works as you can see with the new test cases I added as part of this pull request.

The fact that this is indeed working is irrelevant. There is a clash as you already pointed out and we should better avoid it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works as you can see with the new test cases I added as part of this pull request.

The fact that this is indeed working is irrelevant. There is a clash as you already pointed out and we should better avoid it.

I don't see your point here: the test cases probe that the new build properties whose default values are taken from runtime properties work fine, but you say it's irrelevant?

The current pull request does address and avoids the current linked issue when using flyway / liquibase plus K8s/OpenShift extensions by using well-known properties. It's perfectly fine if you all don't like this solution and/or prefer adding a new build time property that users will need to learn to generate the init-task resources. But don't agree with you saying that the test cases are irrelevant or that the clashing problem is not addressed.

public boolean migrateWithInitTask;
}
Original file line number Diff line number Diff line change
Expand Up @@ -314,13 +314,15 @@ ServiceStartBuildItem startLiquibase(LiquibaseRecorder recorder,
}

@BuildStep
public InitTaskBuildItem configureInitTask() {
return InitTaskBuildItem.create()
.withName("liquibase-init")
.withTaskEnvVars(Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_LIQUIBASE_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_LIQUIBASE_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true);
public void configureInitTask(LiquibaseBuildTimeConfig config, BuildProducer<InitTaskBuildItem> initTasks) {
if (config.generateInitTask && config.defaultDataSource.migrateWithInitTask) {
initTasks.produce(InitTaskBuildItem.create()
.withName("liquibase-init")
.withTaskEnvVars(Map.of("QUARKUS_INIT_AND_EXIT", "true", "QUARKUS_LIQUIBASE_ENABLED", "true"))
.withAppEnvVars(Map.of("QUARKUS_LIQUIBASE_ENABLED", "false"))
.withSharedEnvironment(true)
.withSharedFilesystem(true));
}
}

private Set<String> getDataSourceNames(List<JdbcDataSourceBuildItem> jdbcDataSourceBuildItems) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,15 @@ public LiquibaseDataSourceBuildTimeConfig getConfigForDataSourceName(String data
return namedDataSources.getOrDefault(dataSourceName, LiquibaseDataSourceBuildTimeConfig.defaultConfig());
}

/**
* Flag to enable / disable the generation of the init task Kubernetes resources.
* This property is only relevant if the Quarkus Kubernetes/OpenShift extensions are present.
*
* The default value is `quarkus.liquibase.enabled`.
*/
@ConfigItem(defaultValue = "${quarkus.liquibase.enabled:true}")
public boolean generateInitTask;

/**
* Liquibase configuration for the default datasource.
*/
Expand Down
Loading