Skip to content

Commit

Permalink
Add KubernetesEndpointGroup (#5001)
Browse files Browse the repository at this point in the history
Motivation:

It is tricky to send requests to a Kubernetes cluster from outside
servers without ingress.
There is no way to send traffic directly to the pod, but we can send
traffic to the port of nodes (NodePort) where the pods are located.

This PR proposes a new EndpointGroup that can send requests with CSLB
using NodeIP and NodePort to pods in Kubernetes. This way is not an
ideal CSLB where servers and clients communicate directly, but it will
be a safer way to send traffic without going through ingress which can
be SPOF.

Modifications:

- Add `KubernetesEndpointGroup` on top of `KubernetesClient` to
dynamically obtain Kubernetes resources.
-
[Permission](https://kubernetes.io/docs/reference/access-authn-authz/rbac)
to watch `services`, `nodes`, `pods` is required to fetch endpoints.
- `service.ports[*].nodePort` is used to create the port of `Endpoint`.
- [Watch
API](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes)
is used to track changes in Kubernetes with a minimal delay.
  - `ADDED` and `MODIFIED` events are used to update resouces.
  - `DELETED` is used to remove the resouce.
  - `BOOKMARK` event is not used and `ERROR` may be ignorable.
- Test `KubernetesEndpointGroup` with both a real Kubernetes cluster and
a mock Kubernetes server.

Result:

- You can use `KubernetesEndpointGroup` to perform client-side
load-balancing when sending requests.
- Fixes #4497 
```java
// Create a KubernetesEndpointGroup that fetches the endpoints of the 'my-service' service in the 'default'
// namespace. The Kubernetes client will be created with the default configuration in the $HOME/.kube/config.
KubernetesClient kubernetesClient = new KubernetesClientBuilder().build();
KubernetesEndpointGroup
  .builder(kubernetesClient)
  .namespace("default")
  .serviceName("my-service")
  .build();

// If you want to use a custom configuration, you can create a KubernetesEndpointGroup as follows:
// The custom configuration would be useful when you want to access Kubernetes from outside the cluster.
Config config =
  new ConfigBuilder()
    .withMasterUrl("https://my-k8s-master")
    .withOauthToken("my-token")
    .build();
KubernetesEndpointGroup
  .builder(config)
  .namespace("my-namespace")
  .serviceName("my-service")
  .build();

```
  • Loading branch information
ikhoon authored Apr 9, 2024
1 parent a13d582 commit 388a328
Show file tree
Hide file tree
Showing 12 changed files with 1,100 additions and 4 deletions.
1 change: 1 addition & 0 deletions kubernetes/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ dependencies {
api(libs.kubernetes.client.impl)
testImplementation(variantOf(libs.kubernetes.client.api) { classifier("tests") })
testImplementation(libs.kubernetes.server.mock)
testImplementation(libs.kubernetes.junit.jupiter)
}
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ public CompletableFuture<HttpResponse<AsyncBody>> consumeBytesDirect(
return splitResponse.headers().thenApply(responseHeaders -> {
final AsyncBodySubscriber subscriber = new AsyncBodySubscriber(consumer);
splitResponse.body().subscribe(subscriber, ctx.eventLoop());
return new ArmeriaHttpResponse(responseHeaders, subscriber);
return new ArmeriaHttpResponse(request, responseHeaders, subscriber);
});
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,20 @@
import io.fabric8.kubernetes.client.http.AsyncBody;
import io.fabric8.kubernetes.client.http.HttpRequest;
import io.fabric8.kubernetes.client.http.HttpResponse;
import io.fabric8.kubernetes.client.http.StandardHttpRequest;
import io.netty.util.AsciiString;

final class ArmeriaHttpResponse implements HttpResponse<AsyncBody> {

private final StandardHttpRequest request;
private final ResponseHeaders responseHeaders;
private final AsyncBody body;

@Nullable
private Map<String, List<String>> headers;

ArmeriaHttpResponse(ResponseHeaders responseHeaders, AsyncBody body) {
ArmeriaHttpResponse(StandardHttpRequest request, ResponseHeaders responseHeaders, AsyncBody body) {
this.request = request;
this.responseHeaders = responseHeaders;
this.body = body;
}
Expand All @@ -61,7 +64,7 @@ public AsyncBody body() {

@Override
public HttpRequest request() {
return null;
return request;
}

@Override
Expand Down
Loading

0 comments on commit 388a328

Please sign in to comment.