-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Operator] Add support for Ambassador ingress type
This change adds an Ambassador-based GerritNetworkReconciler, which is used when the `INGRESS` environment variable for the operator is set to "ambassador". In this case, the precondition is that the Ambassador CRDs must already be pre-deployed in the k8s cluster. This change uses the CRD version `getambassador.io/v2`. Ambassador (also known as "Emissary") is an open-source ingress provider that sets up ingresses for services via Custom Resources such as `Mapping`, `TLSContext`, etc. The newly created GerritAmbassadorReconciler creates and manages these resources as josdk "dependent resources". The mappings are created to direct traffic to Primary Gerrit and/or Replica Gerrit and/or Receivers in the GerritCluster. If a GerritCluster has both a Primary and a Replica, then all read traffic (git fetch/clone requests) is directed to the Replica, and all write traffic (git push) is directed to the Primary. Because there does not exist a fabric8 extension for Ambassador custom resources, managing Ambassador CRs in the operator is a little tricky. Two options were considered: 1. Use fabric8 k8s client's `GenericKubernetesResource` class. `GenericKubernetesResource` implements the `HasMetadata` interface, just like the `CustomResource` parent class that is used to define custom resources like GerritCluster etc. `GenericKubernetesResource` can be used to create custom resources by defining the resource spec as a Java Map<String, String>. However, with this option, we would need to subclass `GenericKubernetesResource` (e.g. `OperatorMappingWrapper`) to be able to provide an apiVersion and/or group (expected by josdk). This would introduce an unnecessary CRD to the operator, which is not desirable. 2. Generate Ambassador custom resource POJOs from the CRD yaml using `java-generator-maven-plugin`. This makes it such that it appears to the operator that the Ambassador resources were manually defined in the source code as Java classes, just like the other resources - GerritCluster, Gerrit, etc. We went with option 2. Ambassador CRDs are fetched from https://github.com/emissary-ingress/emissary/blob/master/manifests/emissary/emissary-crds.yaml.in and stored in the repo as a yaml file. The `java-generator-maven-plugin` (fabric8 project) is used to generate POJOs from this CRD yaml. The POJOs represent the Ambassador CRs in Java classes. Manual edits to the Ambassador CRDs - The yaml file defines many CRDs but we only need `Mapping` and `TLSContext` for this change so the rest of the CRDs are deleted from the file - The generator plugin has a bug while converting enum types (fabric8io/kubernetes-client#5457). To avoid hitting this bug, the `v2ExplicitTLS` field in the Mapping CRD `v3alpha1` version is commented out - Emissary CRD apiVersions are not self-contained. There is a field `ambassador_id` in `Mapping` and `TLSContext` CRD that is not defined in apiVersion v2 but users are still able to create v2 Mappings with ambassador_id field (Emissary converts v2 resources to v3 via webhooks defined in emissary service). To be able to define `ambassador_id` in the Mapping/TLSContext CRs created via the operator, we manually add `ambassador_id` to the v2 Mapping and TLSContext CRD. Currently, the operator is watching for changes in resources in all namespaces. If you have Ambassador resources deployed in your k8s cluster that use both v1 and v2 apiVersions, you might run into deserialization errors as the operator attempts to deserialize the existing v1 resources into the v2 POJOs. Change-Id: I23d446e21da87e33c71fe3dad481ec34cd963bbe
- Loading branch information
Showing
36 changed files
with
3,153 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,3 +6,7 @@ | |
| __pycache__ | ||
| .pytest_cache | ||
| *.pyc | ||
|
|
||
| *bin/ | ||
| .DS_Store | ||
| .vscode/ | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 22 additions & 0 deletions
22
operator/src/main/java/com/google/gerrit/k8s/operator/network/Constants.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| // Copyright (C) 2023 The Android Open Source Project | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License.package com.google.gerrit.k8s.operator.network; | ||
|
|
||
| package com.google.gerrit.k8s.operator.network; | ||
|
|
||
| public class Constants { | ||
| public static String UPLOAD_PACK_URL_PATTERN = "/.*/git-upload-pack"; | ||
| public static String INFO_REFS_PATTERN = "/.*/info/refs"; | ||
| public static String RECEIVE_PACK_URL_PATTERN = "/.*/git-receive-pack"; | ||
| public static String PROJECTS_URL_PATTERN = "/a/projects/.*"; | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,5 +17,6 @@ | |
| public enum IngressType { | ||
| NONE, | ||
| INGRESS, | ||
| ISTIO | ||
| ISTIO, | ||
| AMBASSADOR | ||
| } | ||
156 changes: 156 additions & 0 deletions
156
...in/java/com/google/gerrit/k8s/operator/network/ambassador/GerritAmbassadorReconciler.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| // Copyright (C) 2023 The Android Open Source Project | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package com.google.gerrit.k8s.operator.network.ambassador; | ||
|
|
||
| import static com.google.gerrit.k8s.operator.network.ambassador.GerritAmbassadorReconciler.MAPPING_EVENT_SOURCE; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMapping.GERRIT_MAPPING; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingGETReplica.GERRIT_MAPPING_GET_REPLICA; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingPOSTReplica.GERRIT_MAPPING_POST_REPLICA; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingPrimary.GERRIT_MAPPING_PRIMARY; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingReceiver.GERRIT_MAPPING_RECEIVER; | ||
| import static com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingReceiverGET.GERRIT_MAPPING_RECEIVER_GET; | ||
|
|
||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMapping; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingGETReplica; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingPOSTReplica; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingPrimary; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingReceiver; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.GerritClusterMappingReceiverGET; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.LoadBalanceCondition; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.ReceiverMappingCondition; | ||
| import com.google.gerrit.k8s.operator.network.ambassador.dependent.SingleMappingCondition; | ||
| import com.google.gerrit.k8s.operator.network.model.GerritNetwork; | ||
| import com.google.inject.Singleton; | ||
| import io.getambassador.v2.Mapping; | ||
| import io.javaoperatorsdk.operator.api.config.informer.InformerConfiguration; | ||
| import io.javaoperatorsdk.operator.api.reconciler.Context; | ||
| import io.javaoperatorsdk.operator.api.reconciler.ControllerConfiguration; | ||
| import io.javaoperatorsdk.operator.api.reconciler.EventSourceContext; | ||
| import io.javaoperatorsdk.operator.api.reconciler.EventSourceInitializer; | ||
| import io.javaoperatorsdk.operator.api.reconciler.Reconciler; | ||
| import io.javaoperatorsdk.operator.api.reconciler.UpdateControl; | ||
| import io.javaoperatorsdk.operator.api.reconciler.dependent.Dependent; | ||
| import io.javaoperatorsdk.operator.processing.event.source.EventSource; | ||
| import io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource; | ||
| import java.util.HashMap; | ||
| import java.util.Map; | ||
|
|
||
| /** | ||
| * Provides an Ambassador-based implementation for GerritNetworkReconciler. | ||
| * | ||
| * <p>Creates and manages Ambassador Custom Resources using the "managed dependent resources" | ||
| * approach in josdk. Since multiple dependent resources of the same type (`Mapping`) need to be | ||
| * created, "resource discriminators" are used for each of the different Mapping dependent | ||
| * resources. | ||
| * | ||
| * <p>Ambassador custom resource POJOs are generated via the `java-generator-maven-plugin` in the | ||
| * fabric8 project. | ||
| * | ||
| * <p>Mapping logic | ||
| * | ||
| * <p>The Mappings are created based on the composition of Gerrit instances in the GerritCluster. | ||
| * | ||
| * <p>There are three cases: | ||
| * | ||
| * <p>1. 0 Primary 1 Replica | ||
| * | ||
| * <p>Direct all traffic (read/write) to the Replica | ||
| * | ||
| * <p>2. 1 Primary 0 Replica | ||
| * | ||
| * <p>Direct all traffic (read/write) to the Primary | ||
| * | ||
| * <p>3. 1 Primary 1 Replica | ||
| * | ||
| * <p>Direct write traffic to Primary and read traffic to Replica. To capture this requirement, | ||
| * three different Mappings have to be created. | ||
| * | ||
| * <p>Note: git fetch/clone operations result in two HTTP requests to the git server. The first is | ||
| * of the form `GET /my-test-repo/info/refs?service=git-upload-pack` and the second is of the form | ||
| * `POST /my-test-repo/git-upload-pack`. | ||
| * | ||
| * <p>Note: git push operations result in two HTTP requests to the git server. The first is of the | ||
| * form `GET /my-test-repo/info/refs?service=git-receive-pack` and the second is of the form `POST | ||
| * /my-test-repo/git-receive-pack`. | ||
| * | ||
| * <p>If a Receiver is part of the GerritCluster, additional mappings are created such that all | ||
| * requests that the replication plugin sends to the `adminUrl` [1] are routed to the Receiver. This | ||
| * includes `git push` related `GET` and `POST` requests, and requests to the `/projects` REST API | ||
| * endpoints. | ||
| * | ||
| * <p>[1] | ||
| * https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/resources/Documentation/config.md | ||
| */ | ||
| @Singleton | ||
| @ControllerConfiguration( | ||
| // namespaces = "gerrit-operator", | ||
| dependents = { | ||
| @Dependent( | ||
| name = GERRIT_MAPPING, | ||
| type = GerritClusterMapping.class, | ||
| // Cluster has only either Primary or Replica instance | ||
| reconcilePrecondition = SingleMappingCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE), | ||
| @Dependent( | ||
| name = GERRIT_MAPPING_POST_REPLICA, | ||
| type = GerritClusterMappingPOSTReplica.class, | ||
| // Cluster has both Primary and Replica instances | ||
| reconcilePrecondition = LoadBalanceCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE), | ||
| @Dependent( | ||
| name = GERRIT_MAPPING_GET_REPLICA, | ||
| type = GerritClusterMappingGETReplica.class, | ||
| reconcilePrecondition = LoadBalanceCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE), | ||
| @Dependent( | ||
| name = GERRIT_MAPPING_PRIMARY, | ||
| type = GerritClusterMappingPrimary.class, | ||
| reconcilePrecondition = LoadBalanceCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE), | ||
| @Dependent( | ||
| name = GERRIT_MAPPING_RECEIVER, | ||
| type = GerritClusterMappingReceiver.class, | ||
| reconcilePrecondition = ReceiverMappingCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE), | ||
| @Dependent( | ||
| name = GERRIT_MAPPING_RECEIVER_GET, | ||
| type = GerritClusterMappingReceiverGET.class, | ||
| reconcilePrecondition = ReceiverMappingCondition.class, | ||
| useEventSourceWithName = MAPPING_EVENT_SOURCE) | ||
| }) | ||
| public class GerritAmbassadorReconciler | ||
| implements Reconciler<GerritNetwork>, EventSourceInitializer<GerritNetwork> { | ||
|
|
||
| public static final String MAPPING_EVENT_SOURCE = "mapping-event-source"; | ||
|
|
||
| // Because we have multiple dependent resources of the same type `Mapping`, we need to specify | ||
| // a named event source. | ||
| @Override | ||
| public Map<String, EventSource> prepareEventSources(EventSourceContext<GerritNetwork> context) { | ||
| InformerEventSource<Mapping, GerritNetwork> mappingEventSource = | ||
| new InformerEventSource<>( | ||
| InformerConfiguration.from(Mapping.class, context).build(), context); | ||
|
|
||
| Map<String, EventSource> eventSources = new HashMap<>(); | ||
| eventSources.put(MAPPING_EVENT_SOURCE, mappingEventSource); | ||
| return eventSources; | ||
| } | ||
|
|
||
| @Override | ||
| public UpdateControl<GerritNetwork> reconcile( | ||
| GerritNetwork resource, Context<GerritNetwork> context) throws Exception { | ||
| return UpdateControl.noUpdate(); | ||
| } | ||
| } |
62 changes: 62 additions & 0 deletions
62
...gerrit/k8s/operator/network/ambassador/dependent/AbstractAmbassadorDependentResource.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| // Copyright (C) 2023 The Android Open Source Project | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package com.google.gerrit.k8s.operator.network.ambassador.dependent; | ||
|
|
||
| import com.google.gerrit.k8s.operator.cluster.model.GerritCluster; | ||
| import com.google.gerrit.k8s.operator.network.model.GerritNetwork; | ||
| import io.fabric8.kubernetes.api.model.HasMetadata; | ||
| import io.fabric8.kubernetes.api.model.ObjectMeta; | ||
| import io.fabric8.kubernetes.api.model.ObjectMetaBuilder; | ||
| import io.getambassador.v2.MappingSpec; | ||
| import io.getambassador.v2.MappingSpecBuilder; | ||
| import io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDKubernetesDependentResource; | ||
| import java.util.List; | ||
|
|
||
| public abstract class AbstractAmbassadorDependentResource<T extends HasMetadata> | ||
| extends CRUDKubernetesDependentResource<T, GerritNetwork> { | ||
|
|
||
| public AbstractAmbassadorDependentResource(Class<T> dependentResourceClass) { | ||
| super(dependentResourceClass); | ||
| } | ||
|
|
||
| public ObjectMeta getCommonMetadata(GerritNetwork gerritnetwork, String name, String className) { | ||
| ObjectMeta metadata = | ||
| new ObjectMetaBuilder() | ||
| .withName(name) | ||
| .withNamespace(gerritnetwork.getMetadata().getNamespace()) | ||
| .withLabels( | ||
| GerritCluster.getLabels(gerritnetwork.getMetadata().getName(), name, className)) | ||
| .build(); | ||
| return metadata; | ||
| } | ||
|
|
||
| public MappingSpec getCommonSpec(GerritNetwork gerritnetwork, String serviceName) { | ||
| MappingSpec spec = | ||
| new MappingSpecBuilder() | ||
| .withAmbassadorId(getAmbassadorIds(gerritnetwork)) | ||
| .withHost(gerritnetwork.getSpec().getIngress().getHost()) | ||
| .withPrefix("/") | ||
| .withService(serviceName) | ||
| .withBypassAuth(true) | ||
| .withRewrite("") // important - so the prefix doesn't get overwritten to "/" | ||
| .build(); | ||
| return spec; | ||
| } | ||
|
|
||
| public List<String> getAmbassadorIds(GerritNetwork gerritnetwork) { | ||
| // TODO: Allow users to configure ambassador_id | ||
| return null; | ||
| } | ||
| } |
Oops, something went wrong.