Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve usability of gRPC probes in Kubernetes #33437

Merged
merged 1 commit into from
May 26, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/src/main/asciidoc/deploying-to-kubernetes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1281,6 +1281,7 @@ quarkus.kubernetes.env-vars.foo.configmap=myconfigmap
`quarkus.kubernetes.env-vars` are deprecated (though still currently supported as of this writing) and the new declaration style should be used instead.
See <<#env-vars>> and more specifically <<env-vars-backwards>> for more details.

[[deployment]]
== Deployment

To trigger building and deploying a container image you need to enable the `quarkus.kubernetes.deploy` flag (the flag is disabled by default - furthermore it has no effect during test runs or dev mode).
Expand Down
26 changes: 20 additions & 6 deletions docs/src/main/asciidoc/grpc-getting-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,29 +25,43 @@ The solution is located in the `grpc-plain-text-quickstart` {quickstarts-tree-ur

== Configuring your project

Edit the `pom.xml` file to add the Quarkus gRPC extension dependency (just under `<dependencies>`):
Add the Quarkus gRPC extension to your build file:

[source,xml]
[source,xml,role="primary asciidoc-tabs-target-sync-cli asciidoc-tabs-target-sync-maven"]
.pom.xml
----
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-grpc</artifactId>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-grpc</artifactId>
</dependency>
----

[source,gradle,role="secondary asciidoc-tabs-target-sync-gradle"]
.build.gradle
----
implementation("io.quarkus:quarkus-grpc")
----

By default, the `quarkus-grpc` extension relies on the reactive programming model.
In this guide we will follow a reactive approach.
Under the `dependencies` section of your `pom.xml` file, make sure you have the RESTEasy Reactive dependency:

[source,xml]
[source,xml,role="primary asciidoc-tabs-target-sync-cli asciidoc-tabs-target-sync-maven"]
.pom.xml
----
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy-reactive</artifactId>
</dependency>
----

Make sure you have `generate-code` goal of `quarkus-maven-plugin` enabled in your `pom.xml`.
[source,gradle,role="secondary asciidoc-tabs-target-sync-gradle"]
.build.gradle
----
implementation("io.quarkus:quarkus-resteasy-reactive")
----

If you are using Maven, make sure you have the `generate-code` goal of `quarkus-maven-plugin` enabled in your `pom.xml`.
If you wish to generate code from different `proto` files for tests, also add the `generate-code-tests` goal.
Please note that no additional task/goal is required for the Gradle plugin.

Expand Down
92 changes: 92 additions & 0 deletions docs/src/main/asciidoc/grpc-kubernetes.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
////
This guide is maintained in the main Quarkus repository
and pull requests should be submitted there:
https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
////
= Deploying your gRPC Service in Kubernetes
include::_attributes.adoc[]
:categories: serialization
:summary: This guide explains how to deploy your gRPC services in Quarkus to Kubernetes.

This page explains how to deploy your gRPC service in Quarkus in Kubernetes.
We'll continue with the example from xref:grpc-getting-started.adoc[the Getting Started gRPC guide].

== Configuring your project to use the Quarkus Kubernetes extension

Add the Quarkus Kubernetes extension to your build file:

[source,xml,role="primary asciidoc-tabs-target-sync-cli asciidoc-tabs-target-sync-maven"]
.pom.xml
----
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-kubernetes</artifactId>
</dependency>
----

[source,gradle,role="secondary asciidoc-tabs-target-sync-gradle"]
.build.gradle
----
implementation("io.quarkus:quarkus-kubernetes")
----

Next, we want to expose our application using the Kubernetes Ingress resource:

[source,properties]
----
quarkus.kubernetes.ingress.expose=true
----

The Quarkus Kubernetes will bind the HTTP server using the port name `http` and the gRPC server using the port name `grpc`. By default, the Quarkus application will only expose the port name `http`, so only the HTTP server will be publicly accessible. To expose the gRPC server instead, set the `quarkus.kubernetes.ingress.target-port=grpc` property in your application.properties:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does that work with the new server where both http and gRPC are served on the same port?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you share more details about this? But I guess that if both HTTP and gRPC are handing by the same port, it will work fine using the default http port name.
Though, if the port name grpc won't be generated any longer, we might need to revisit this statement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the old gRPC server, http and gRPC are served on different ports.
With the new gRPC server, HTTP and gRPC are served on the same port (which is very likely going to be https).

Both mode are supported at the moment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alesj do you think we could add tests for both cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a deeper look into this that was implemented as part of #28654.
So, by default, it creates two servers: one for HTTP and another one for gRPC.
If quarkus.grpc.server.use-separate-server is false, then the gRPC port won't be listening.

Note that the grpc port is still bound to the generated Deployment/Service resources even though quarkus.grpc.server.use-separate-server is false. Because quarkus.grpc.server.use-separate-server is a runtime property, this might make sense. So, ultimately the user would need to select the target port (either http or grpc) of the generated resource.

We could again follow a best-effort approach, so if users use quarkus.grpc.server.use-separate-server=false at build time, we could remove the binding of the grpc port in the generated resources and use http by default as the target port. Wdyt @cescoffier ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 or having an explicit property on the kubernetes side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have an explicit property on K8s side which is quarkus.kubernetes.ingress.target-port.
I'd prefer doing We could again follow a best-effort approach, so if users use quarkus.grpc.server.use-separate-server=false at build time, we could remove the binding of the grpc port in the generated resources and use http by default as the target port. as part of another pull request after merging this pull request.

So, if you're ok with the rest of the changes, can we proceed with merging this pull request?

[source,properties]
----
quarkus.kubernetes.ingress.target-port=grpc
----

TIP: If you configure Quarkus to use the same port for both HTTP and gRPC servers with the property `quarkus.grpc.server.use-separate-server=false`, then you don't need to change the default `target-port`.

Finally, we need to generate the Kubernetes manifests by running the command in a terminal:

include::{includes}/devtools/build.adoc[]

Once generated, you can look at the `target/kubernetes` directory:

[source,txt]
----
target/kubernetes
└── kubernetes.json
└── kubernetes.yml
----

You can find more information about how to deploy the application in Kubernetes in the xref:deploying-to-kubernetes.adoc#deployment[the Kubernetes guide].

== Using gRPC Health probes

By default, the Kubernetes resources do not contain readiness and liveness probes. To add them, import the Smallrye Health extension to your build file:

[source,xml,role="primary asciidoc-tabs-target-sync-cli asciidoc-tabs-target-sync-maven"]
.pom.xml
----
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-health</artifactId>
</dependency>
----

[source,gradle,role="secondary asciidoc-tabs-target-sync-gradle"]
.build.gradle
----
implementation("io.quarkus:quarkus-smallrye-health")
----

TIP: More information about the health extension can be found in xref:microprofile-health.adoc[the Microprofile Health guide].

By the default, this extension will configure the probes to use the HTTP server (which is provided by some extensions like the Quarkus RESTEasy reactive extension). Internally, this probe will also use xref:grpc-service-implementation.adoc#health[the generated gRPC Health services].

If your application does not use any Quarkus extension that exposes an HTTP server, you can still configure the probes to directly use the gRPC Health service by adding the property `quarkus.kubernetes.readiness-probe.grpc-action-enabled=true` into your configuration:

[source,properties]
----
quarkus.kubernetes.readiness-probe.grpc-action-enabled=true
----
1 change: 1 addition & 0 deletions docs/src/main/asciidoc/grpc-service-implementation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ public class StreamingService implements Streaming {
}
----

[[health]]
== Health Check
For the implemented services, Quarkus gRPC exposes health information in the following format:
[source,protobuf]
Expand Down
15 changes: 11 additions & 4 deletions docs/src/main/asciidoc/grpc-xds.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,23 @@ with shaded grpc-netty library while running native IT tests.

== Configuring your project

Edit the `pom.xml` file to add the Quarkus gRPC xDS dependency (just under `<dependencies>`):
Add the Quarkus gRPC xDS extension to your build file:

[source,xml]
[source,xml,role="primary asciidoc-tabs-target-sync-cli asciidoc-tabs-target-sync-maven"]
.pom.xml
----
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-grpc-xds</artifactId>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-grpc-xds</artifactId>
</dependency>
----

[source,gradle,role="secondary asciidoc-tabs-target-sync-gradle"]
.build.gradle
----
implementation("io.quarkus:quarkus-grpc-xds")
----

NOTE: This transitively adds `io.quarkus:quarkus-grpc` extension dependency.

== Server configuration
Expand Down
1 change: 1 addition & 0 deletions docs/src/main/asciidoc/grpc.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,5 @@ Quarkus gRPC is based on https://vertx.io/docs/vertx-grpc/java/[Vert.x gRPC].
* xref:grpc-getting-started.adoc[Getting Started]
* xref:grpc-service-implementation.adoc[Implementing a gRPC Service]
* xref:grpc-service-consumption.adoc[Consuming a gRPC Service]
* xref:grpc-kubernetes.adoc[Deploying your gRPC Service in Kubernetes]
* xref:grpc-xds.adoc[Enabling xDS gRPC support]
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ private static ContainerBuilder convert(String name, ContainerConfig c) {
c.command.ifPresent(w -> b.withCommand(w.toArray(new String[0])));
c.arguments.ifPresent(w -> b.withArguments(w.toArray(new String[0])));
if (c.readinessProbe != null && c.readinessProbe.hasUserSuppliedAction()) {
b.withReadinessProbe(ProbeConverter.convert(c.readinessProbe));
b.withReadinessProbe(ProbeConverter.convert(name, c.readinessProbe));
}
if (c.livenessProbe != null && c.livenessProbe.hasUserSuppliedAction()) {
b.withLivenessProbe(ProbeConverter.convert(c.livenessProbe));
b.withLivenessProbe(ProbeConverter.convert(name, c.livenessProbe));
}
b.addAllToEnvVars(c.convertToEnvs());
c.ports.entrySet().forEach(e -> b.addToPorts(PortConverter.convert(e)));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -985,10 +985,11 @@ private static Optional<DecoratorBuildItem> createLivenessProbe(String name, Str
Optional<KubernetesHealthLivenessPathBuildItem> livenessPath) {
if (livenessProbe.hasUserSuppliedAction()) {
return Optional.of(
new DecoratorBuildItem(target, new AddLivenessProbeDecorator(name, ProbeConverter.convert(livenessProbe))));
new DecoratorBuildItem(target,
new AddLivenessProbeDecorator(name, ProbeConverter.convert(name, livenessProbe))));
} else if (livenessPath.isPresent()) {
return Optional.of(new DecoratorBuildItem(target, new AddLivenessProbeDecorator(name,
ProbeConverter.builder(livenessProbe).withHttpActionPath(livenessPath.get().getPath()).build())));
ProbeConverter.builder(name, livenessProbe).withHttpActionPath(livenessPath.get().getPath()).build())));
}
return Optional.empty();
}
Expand All @@ -997,10 +998,10 @@ private static Optional<DecoratorBuildItem> createReadinessProbe(String name, St
Optional<KubernetesHealthReadinessPathBuildItem> readinessPath) {
if (readinessProbe.hasUserSuppliedAction()) {
return Optional.of(new DecoratorBuildItem(target,
new AddReadinessProbeDecorator(name, ProbeConverter.convert(readinessProbe))));
new AddReadinessProbeDecorator(name, ProbeConverter.convert(name, readinessProbe))));
} else if (readinessPath.isPresent()) {
return Optional.of(new DecoratorBuildItem(target, new AddReadinessProbeDecorator(name,
ProbeConverter.builder(readinessProbe).withHttpActionPath(readinessPath.get().getPath()).build())));
ProbeConverter.builder(name, readinessProbe).withHttpActionPath(readinessPath.get().getPath()).build())));
}
return Optional.empty();
}
Expand All @@ -1009,10 +1010,10 @@ private static Optional<DecoratorBuildItem> createStartupProbe(String name, Stri
Optional<KubernetesHealthStartupPathBuildItem> startupPath) {
if (startupProbe.hasUserSuppliedAction()) {
return Optional.of(new DecoratorBuildItem(target,
new AddStartupProbeDecorator(name, ProbeConverter.convert(startupProbe))));
new AddStartupProbeDecorator(name, ProbeConverter.convert(name, startupProbe))));
} else if (startupPath.isPresent()) {
return Optional.of(new DecoratorBuildItem(target, new AddStartupProbeDecorator(name,
ProbeConverter.builder(startupProbe).withHttpActionPath(startupPath.get().getPath()).build())));
ProbeConverter.builder(name, startupProbe).withHttpActionPath(startupPath.get().getPath()).build())));
}
return Optional.empty();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ public class ProbeConfig {
@ConfigItem
Optional<String> grpcAction;

/**
* If enabled and `grpc-action` is not provided, it will use the generated service name and the gRPC port.
*/
@ConfigItem(defaultValue = "false")
boolean grpcActionEnabled;
Sgitario marked this conversation as resolved.
Show resolved Hide resolved

/**
* The amount of time to wait before starting to probe.
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,38 @@

package io.quarkus.kubernetes.deployment;

import org.eclipse.microprofile.config.ConfigProvider;

import io.dekorate.kubernetes.config.Probe;
import io.dekorate.kubernetes.config.ProbeBuilder;

public class ProbeConverter {

public static Probe convert(ProbeConfig probe) {
return builder(probe).build();
public static Probe convert(String name, ProbeConfig probe) {
return builder(name, probe).build();
}

public static ProbeBuilder builder(ProbeConfig probe) {
public static ProbeBuilder builder(String name, ProbeConfig probe) {
ProbeBuilder b = new ProbeBuilder();
probe.httpActionPath.ifPresent(b::withHttpActionPath);
probe.execAction.ifPresent(b::withExecAction);
probe.tcpSocketAction.ifPresent(b::withTcpSocketAction);
probe.grpcAction.ifPresent(b::withGrpcAction);
if (probe.grpcAction.isPresent()) {
b.withGrpcAction(probe.grpcAction.get());
} else if (probe.grpcActionEnabled) {
b.withGrpcAction(getQuarkusGrpcPort() + ":" + name);
}

b.withInitialDelaySeconds((int) probe.initialDelay.getSeconds());
b.withPeriodSeconds((int) probe.period.getSeconds());
b.withTimeoutSeconds((int) probe.timeout.getSeconds());
b.withSuccessThreshold(probe.successThreshold);
b.withFailureThreshold(probe.failureThreshold);
return b;
}

private static int getQuarkusGrpcPort() {
return ConfigProvider.getConfig().getOptionalValue("quarkus.grpc.server.port", Integer.class)
.orElse(9000);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
package io.quarkus.it.kubernetes;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertEquals;

import java.io.IOException;
import java.nio.file.Path;
import java.util.List;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.fabric8.kubernetes.api.model.apps.Deployment;
import io.quarkus.builder.Version;
import io.quarkus.maven.dependency.Dependency;
import io.quarkus.test.ProdBuildResults;
import io.quarkus.test.ProdModeTestResults;
import io.quarkus.test.QuarkusProdModeTest;

public class KubernetesWithGrpcProbeEnabledTest {

private static final String APP_NAME = "kubernetes-with-grpc-probe";

@RegisterExtension
static final QuarkusProdModeTest config = new QuarkusProdModeTest()
.withApplicationRoot((jar) -> jar.addClasses(GreetingResource.class))
.setApplicationName(APP_NAME)
.setApplicationVersion("0.1-SNAPSHOT")
.overrideConfigKey("quarkus.kubernetes.readiness-probe.grpc-action-enabled", "true")
.setLogFileName("k8s.log")
.setForcedDependencies(List.of(
Dependency.of("io.quarkus", "quarkus-kubernetes", Version.getVersion()),
Dependency.of("io.quarkus", "quarkus-smallrye-health", Version.getVersion())));

@ProdBuildResults
private ProdModeTestResults prodModeTestResults;

@Test
public void assertGeneratedResources() throws IOException {

final Path kubernetesDir = prodModeTestResults.getBuildDir().resolve("kubernetes");
assertThat(kubernetesDir)
.isDirectoryContaining(p -> p.getFileName().endsWith("kubernetes.json"))
.isDirectoryContaining(p -> p.getFileName().endsWith("kubernetes.yml"));
List<HasMetadata> kubernetesList = DeserializationUtil
.deserializeAsList(kubernetesDir.resolve("kubernetes.yml"));
assertThat(kubernetesList.get(0)).isInstanceOfSatisfying(Deployment.class, d -> {
assertThat(d.getMetadata()).satisfies(m -> {
assertThat(m.getName()).isEqualTo(APP_NAME);
});

assertThat(d.getSpec()).satisfies(deploymentSpec -> {
assertThat(deploymentSpec.getTemplate()).satisfies(t -> {
assertThat(t.getSpec()).satisfies(podSpec -> {
assertThat(podSpec.getContainers()).singleElement()
.satisfies(container -> {
assertThat(container.getReadinessProbe()).isNotNull().satisfies(p -> {
assertEquals(p.getGrpc().getPort().intValue(), 9000);
assertEquals(p.getGrpc().getService(), APP_NAME);
});
});
});
});
});
});
}
}