From 04564e3d705a425227561fbc76fe996597e22740 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Fri, 3 Jan 2020 22:12:01 +0100 Subject: [PATCH 01/17] First draft for review of the KEP for Mixed protocol support in Services --- .../sig-network/20200103-mixed-protocol-lb.md | 351 ++++++++++++++++++ 1 file changed, 351 insertions(+) create mode 100644 keps/sig-network/20200103-mixed-protocol-lb.md diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md new file mode 100644 index 00000000000..f8fcdd406b7 --- /dev/null +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -0,0 +1,351 @@ +--- +title: Different protocols in the same Service definition with type=LoadBalancer +authors: + - "@janosi" +owning-sig: sig-network +participating-sigs: + - sig-cloud-provider +reviewers: + - "@thockin" + - "@dcbw" + - "@andrewsykim" +approvers: + - "@thockin" +editor: TBD +creation-date: 2020-01-03 +last-updated: 2020-01-03 +status: provisional +see-also: +replaces: +superseded-by: +--- + +# different protocols in the same service definition with type=loadbalancer + +## Table of Contents + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories [optional]](#user-stories-optional) + - [Story 1](#story-1) + - [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional) + - [Alibaba](#alibaba) + - [AWS](#aws) + - [Azure](#azure) + - [GCE](#gce) + - [IBM Cloud](#ibm-cloud) + - [OpenStack](#openstack) + - [Oracle Cloud](#oracle-cloud) + - [Tencent Cloud](#tencent-cloud) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Option Control Alternatives](#option-control-alternatives) + - [Annotiation in the Service definition](#annotiation-in-the-service-definition) + - [Merging Services in CPI](#merging-services-in-cpi) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [Examples](#examples) + - [Alpha -> Beta Graduation](#alpha---beta-graduation) + - [Beta -> GA Graduation](#beta---ga-graduation) + - [Removing a deprecated flag](#removing-a-deprecated-flag) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Implementation History](#implementation-history) +- [Drawbacks [optional]](#drawbacks-optional) +- [Alternatives [optional]](#alternatives-optional) +- [Infrastructure Needed [optional]](#infrastructure-needed-optional) + + +## Release Signoff Checklist + +- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR) https://github.com/kubernetes/enhancements/issues/1435 +- [ ] KEP approvers have set the KEP status to `implementable` +- [ ] Design details are appropriately documented +- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [ ] Graduation criteria is in place +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + +## Summary + +This feature enables the creation of a LoadBalancer Service that has different port definitions with different protocols. + +## Motivation + +The ultimate goal of this feature is to support users that want to expose their applications via a single IP address but different L4 protocols with a cloud provided load-balancer. +The following issue and PR shows considerable interest from users that would benefit from this feature: +https://github.com/kubernetes/kubernetes/issues/23880 +https://github.com/kubernetes/kubernetes/pull/75831 + +### Goals + +The goals of this KEP are: + +- To analyze the impact of this feature with regard to current implementations of cloud-provider load-balancers +- define how the activation of this feature could be made configurable if certain cloud-provider load-balancer implementations do not want to provide this feature + +### Non-Goals + + +## Proposal + +The first thing proposed here is to lift the hardcoded limitation in Kubernetes that currently rejects Service definitions with different protocols if their type is LoadBalancer. Kubernetes would not reject Service definitions like this from that point: +```yaml +apiVersion: v1 +kind: Service +metadata: + name: mixed-protocol +spec: + type: LoadBalancer + ports: + - name: dns-udp + port: 53 + protocol: UDP + - name: dns-tcp + port: 53 + protocol: TCP + selector: + app: my-dns-server + ``` + +Once that limit is removed those Service definitions will reach the Cloud Provider LB controller implementations. The logic of the particular Cloud Provider LB controller and of course the actual capabilities and architecture of the backing Cloud Load Balancer services determines how the actual exposure of the application really manifests. For this reason it is important to understand the capabilities of those backing services and to design this feature accordingly. + +### User Stories [optional] + +#### Story 1 + +As a Kubernetes cluster user I want to expose an application that provides its service on different protocols with a single cloud provider load balancer IP. In order to achieve this I want to define different `ports` with mixed protocols in my Service definition of `type: LoadBalancer` + +### Implementation Details/Notes/Constraints [optional] + +#### Alibaba + +The Alibaba Cloud Provider Interface Implementation (CPI) supports TCP, UDP and HTTPS in Service definitions and can configure the SLB listeners with the protocols defined in the Service. + +The Alibaba SLB supports TCP, UDP and HTTPS listeners. A listener must be configured for each protocol, and then those listeners can be assigned to the same SLB instance. + +The number of listeners does not affect SLB pricing. +https://www.alibabacloud.com/help/doc-detail/74809.htm + +A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"`. Internal SLBs are free. + +Summary: Alibaba CPI and SLB seems to support mixed protocol Services, and the pricing is not based on the number of protocols per Service. + +#### AWS + +The AWS CPI does not support mixed protocols in the Service definition since it only allows TCP for load balancers. The AWS CPI looks for annotations on the Service to determine whether TCP, TLS or HTTP(S) listener should be created in the AWS ELB for a configured Service port. + +AWS Classic LB supports TCP,TLS, and HTTP(S) protocols behind the same IP address. + +AWS Network LB supports TCP/TLS and UDP protocols behind the same IP address. As we can see, UDP cannot be utilized currently, due to the limitation in the AWS CPI. + +The usage of TCP+HTTP or UDP+HTTP on the same LB instace behind the same IP address is not possible in AWS. + +From a pricing perspective the AWS NLB and the CLB have the following models: +https://aws.amazon.com/elasticloadbalancing/pricing/ +Both are primarily usage based rather than charging based on the number of protocol, however NLBs have separate pricing unit quotas for TCP, UDP and TLS. + +A user can ask for an internal Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0`. So far the author could not find any difference in the usage and pricing of those when compared to the external LBs - except the pre-requisite of a private subnet on which the LB can be deployed. + +Summary: AWS CPI is the current bottleneck with its TCP-only limitation. As long as it is there the implementation of this feature will have no effect on the AWS bills. + +#### Azure + +Azure CPI LB documentation: https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/docs/services/README.md + +The Azure CPI already supports the usage of both UDP and TCP protocols in the same Service definition. It is achieved with the CPI specific annotation `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols`. If this key has value of `true` in the Service definition, the Azure CPI adds the other protocol value (UDP or TCP) to its internal Service representation. Consequently, it also manages twice the amount of load balancer rules for the specific frontend. + +Only TCP and UDP are supported in the current mixed protocol configuration. + +The Azure Load Balancer supports only TCP and UDP as protocols. HTTP support would require the usage of the Azure Application Gateway. I.e. HTTP and L4 protocols cannot be used on the same LB instance/IP address + +Basic Azure Load Balancers are free. + +Pricing of the Standard Azure Load Balancer is based on load-balancing rules and outbound rules. There is a flat price up to 5 rules, on top of which every new forwarding rule has additional cost. +https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#pricing + +A user can ask for an internal Azure Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/azure-load-balancer-internal: true`. There is not any limitation on the usage of mixed service protocols with internal LBs in the Azure CPI. + +Summary: Azure already has mixed protocol support for TCP and UDP, and it already affects the bills of the users. The implementation of this feature may require some work in Azure CPI. + +#### GCE + +The GCE CPI supports only TCP and UDP protocols in Services. + +GCE/GKE creates Network Load Balancers based on the K8s Services with type=LoadBalancer. In GCE there are "forwarding rules" that define how the incoming traffic shall be forwarded to the compute instances. A single forwarding rule can support either TCP or UDP but not both. In order to have both TCP and UDP forwarding rules we have to create separate forwarding rule instances for those. Two or more forwarding rules can share the same external IP if +- the network load balancer type is External +- the external IP address is not ephemeral but static + +There is a workaround in GCE, please see this comment from the original issue: https://github.com/kubernetes/kubernetes/issues/23880#issuecomment-269054735 If the external IP is static the user can create two Service instances, one for UDP and another for TCP and the user has to specify that static external IP address in those Service definitions in `loadBalancerIP`. + +HTTP protocol support: there is a different LB type in GCE for HTTP traffic: HTTP(S) LB. I.e. just like in the case of AWS it is not possible to have e.g. TCP+HTTP or UDP+HTTP behind the same LB/IP address. + +Forwarding rule pricing is per rule: there is a flat price up to 5 forwarding rule instances, on top of which every new forwarding rule has additional cost. + +https://cloud.google.com/compute/network-pricing#forwarding_rules_charges + +[GCP forwarding_rules_charges](https://cloud.google.com/compute/network-pricing#forwarding_rules_charges) suggest that the same K8s Service definition would result in the creation of 2 forwarding rules in GCP. This has the same fixed price up to 5 forwarding rule instances, and each additional rule results in extra cost. + +A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `cloud.google.com/load-balancer-type: "Internal"`. Forwarding rules are also part of the GCE Internal TCP/UDP Load Balancer architecture, but in case of Internal TCP/UDP Load Balancer it is not supported to define different forwarding rules with different protocols for the same IP address. That is, for Services with type=LoadBalancer and with annotation `cloud.google.com/load-balancer-type: "Internal"` this feature would not be supported. + +Summary: The implementation of this feature can affect the bills of GCP users. However the following perspectives are also observed: +- if a user wants to have UDP and TCP ports behind the same NLB two Services with must be defined, one for TCP and one for UDP. As the pricing is based on the number of forwarding rules this setup also means the same pricing as with the single Service instance. +- if a user is happy with 2 NLB (Service) instances for TCP and UDP still the user has two more forwarding rules to be billed - i.e. it has the same effect on pricing as if those TCP and UDP endpoints were behind the same NLB instance +- already now the bills of users is affected if they have more than 5 forwarding rules as the result of their current Service definitions (e.g. 6 or more port definitions in a single Serice, or if the number of all port definitions in different Services is 6 or more, etc.) + +That is, if we consider the "single Service with 6 ports" case the user has to pay more for that single Service ("single LB instance") than for another Service (another LB instance) with 5 or less ports already now. It is not the number of LBs that matters. This phenomenon is already there with the current practice, and the enabling of mixed protocols will not change it to the worse. + +#### IBM Cloud + +The IBM Cloud CPI implementation supports TCP, UDP, HTTP protocol values in K8s Services, and it supports mutiple protocols in a Service. The IBM Cloud CPI creates VPC Load Balancer in a VPC based cluster and NLB in a classic cloud based cluster. + +The VPC Load Balancer supports TCP and HTTP, and it is possible to create TCP and HTTP listeners for the same LB instance. UDP is not supported. + +The VPC LB pricing is time and data amount based, i.e. the number of protocols on the same LB instance does not affect it. + +The NLB supports TCP and UDP on the same NLB instance. The usage of NLB does not have pricing effects, it is part of the IKS basic package. + +Summary: once this feature is implemented IBM Cloud VPC LB can use TCP and HTTP ports from a single Service definition. NLB can use TCP and UDP ports from a single Service definition. + +#### OpenStack + +The OpenStack CPI supports TCP, UDP, HTTP(S) in Service definitions and can configure the Octavia listeners with the protocols defined in the Service. +OpenStack Octavia supports TCP, UDP and HTTP(S) on listeners, an own listener must be configured for each protocol, and different listeners can be used on the same LB instance. + +Summary: the OpenStack based clouds that use Octavia v2 as their LBaaS seems to support this feature once implemented. Pricing is up to their model. + +#### Oracle Cloud + +Oracle Cloud supports TCP, HTTP(S) protocols in its LB solution. The Oracle CPI also supports the protocols in he K8s Service definitions. + +The pricing is based on time and capacity: https://www.oracle.com/cloud/networking/load-balancing-pricing.html I.e. the amount of protocols on a sinlge LB instance does not affect the pricing. + +#### Tencent Cloud + +The Tencent Cloud CPI supports TCP, UDP and HTTP(S) in Service definitions and can configure the CLB listeners with the protocols defined in the Service. +The Tencent Cloud CLB supports TCP, UDP and HTTP(S) listeners, an own listener must be configured for each protocol. +The number of listeners does not affect CLB pricing. CLB pricing is time (day) based and not tied to the number of listeners. +https://intl.cloud.tencent.com/document/product/214/8848 + +A user can ask for an internal Load Balancer via a K8s Service definition that has the annotation `service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: subnet-xxxxxxxx`. Internal CLBs are free. + + +### Risks and Mitigations + +The goal of the current restriction on the K8s API was to prevent an unexpected extra charging for Load Balancers that were created based on Services with mixed protocol definitions. +If the current limitation is removed without any option control we introduce the same risk. Let's see which clouds are exposed: +- Alibaba: the pricing here is not protocol or forwarding rule or listener based. No risk. +- AWS: there is no immediate impact on the pricing side as the AWS CPI limits the scope of protocols to TCP only. +- Azure: Azure pricing is indeed based on load-balancing rules, but this cloud already supports mixed protocols via annotations. There is another risk for Azure, though: if the current restriction is removed from the K8s API, the Azure CPI must be prepared to handle Services with mixed protocols. +- GCE: here the risk is valid once the user exceeds the threshold of 5 forwarding rules. Though, as we mentioned above, it is already possible now without this feature +- IBM Cloud: no risk +- OpenStack: here the risk is that there is almost no chance to analyze all the public OpenStack cloud providers with regard to their pricing policies +- Oracle: no risk +- Tencent Cloud: no risk + +The other risk is in the introduction of this feature without an option control mechanism, i.e. as a general change in Service handling. In that case there is the question whether this feature should be a part of the conformance test set, because it can affect the conformance of cloud providers. + +A possible mitigation is to put the feature behind option control. + +## Design Details + +The implementation of the basic logic is ready in this PR: +https://github.com/kubernetes/kubernetes/pull/75831 + +Currently a feature gate is used to control its activation status. Though if we want to keep this feature behind option control even after it reaches its GA state we should come up with a different solution, as feature gates are used to control the status of a feature as that graduates from alpha to GA, and they are not meant for option control for features with GA status. + +### Option Control Alternatives + +#### Annotiation in the Service definition + +In this alternative we would have a new annotation in the `kubernetes.io` annotation namespace, as it was planned in the original PR. Example: `kubernetes.io/mixedprotocol`. If this annotation is applied on a Service definition the Service would be accepted by the K8s API. + +Pro: +- a kind of "implied conduct" from the user's side. The user explicitly defines with the usage of the annotation that the usage of multiple protocols on the same LoadBalancer is accepted +- Immediate feedback from the K8s API if the user configures a Service with mixed protocol set but without this annotation +Con: +- Additional configuration task for the user +- Must be executed for each and every Service definitions that are to define different protocols for the same LB + +#### Merging Services in CPI + +This one is not really a classic option control mechanism. The idea comes from the current practice implemented in MetalLB: https://metallb.universe.tf/usage/#ip-address-sharing +The same works in GCE as a workaround. + +I.e. if a cloud provider wants to support this feature the CPI must have a logic to apply the Service definitions with a common key value (for example loadBalancerIP) on the same LoadBalancer instance. If the CPI does not implement this support it will work as it does currently. +Pro: +- the cloud provider can decide when to support this feature, and until that it works as currently for this kind of config +Con: +- the users must maintain two Service instances +- Atomic update can be a problem - the key must be such a Service attribute that cannot be patched + +### Test Plan + +### Graduation Criteria + +#### Examples + +##### Alpha -> Beta Graduation + +- Gather feedback from developers and surveys +- Complete features A, B, C +- Tests are in Testgrid and linked in KEP + +##### Beta -> GA Graduation + +- N examples of real world usage +- N installs +- More rigorous forms of testing e.g., downgrade tests and scalability tests +- Allowing time for feedback + +##### Removing a deprecated flag + +- Announce deprecation and support policy of the existing flag +- Two versions passed since introducing the functionality which deprecates the flag (to address version skew) +- Address feedback on usage/changed behavior, provided on GitHub issues +- Deprecate the flag + +**For non-optional features moving to GA, the graduation criteria must include [conformance tests].** + +[conformance tests]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md + +### Upgrade / Downgrade Strategy + +If applicable, how will the component be upgraded and downgraded? Make sure this is in the test plan. + +Consider the following in developing an upgrade/downgrade strategy for this enhancement: +- What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade in order to keep previous behavior? +- What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade in order to make use of the enhancement? + +### Version Skew Strategy + +If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure +this is in the test plan. + +Consider the following in developing a version skew strategy for this enhancement: +- Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used? +- Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet. + +## Implementation History + +Major milestones in the life cycle of a KEP should be tracked in `Implementation History`. +Major milestones might include + +- the `Summary` and `Motivation` sections being merged signaling SIG acceptance +- the `Proposal` section being merged signaling agreement on a proposed design +- the date implementation started +- the first Kubernetes release where an initial version of the KEP was available +- the version of Kubernetes where the KEP graduated to general availability +- when the KEP was retired or superseded + +## Drawbacks [optional] + +## Alternatives [optional] + +## Infrastructure Needed [optional] From fbc0b615c716f0747ddbcd7bcbef9f3271d679ff Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Thu, 9 Jan 2020 22:26:31 +0100 Subject: [PATCH 02/17] Fix bad spelling and remove template parts based on review comment --- keps/sig-network/20200103-mixed-protocol-lb.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index f8fcdd406b7..f6d569b5ea2 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -116,13 +116,13 @@ spec: Once that limit is removed those Service definitions will reach the Cloud Provider LB controller implementations. The logic of the particular Cloud Provider LB controller and of course the actual capabilities and architecture of the backing Cloud Load Balancer services determines how the actual exposure of the application really manifests. For this reason it is important to understand the capabilities of those backing services and to design this feature accordingly. -### User Stories [optional] +### User Stories #### Story 1 As a Kubernetes cluster user I want to expose an application that provides its service on different protocols with a single cloud provider load balancer IP. In order to achieve this I want to define different `ports` with mixed protocols in my Service definition of `type: LoadBalancer` -### Implementation Details/Notes/Constraints [optional] +### Implementation Details/Notes/Constraints #### Alibaba @@ -262,7 +262,7 @@ Currently a feature gate is used to control its activation status. Though if we ### Option Control Alternatives -#### Annotiation in the Service definition +#### Annotation in the Service definition In this alternative we would have a new annotation in the `kubernetes.io` annotation namespace, as it was planned in the original PR. Example: `kubernetes.io/mixedprotocol`. If this annotation is applied on a Service definition the Service would be accepted by the K8s API. From cceb5acfd416c320deea45ed0610b7677049045c Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Wed, 15 Jan 2020 12:28:47 +0100 Subject: [PATCH 03/17] Update OpenStack part with the appropriate Octavia version --- keps/sig-network/20200103-mixed-protocol-lb.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index f6d569b5ea2..e7538beac2d 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -218,7 +218,10 @@ Summary: once this feature is implemented IBM Cloud VPC LB can use TCP and HTTP The OpenStack CPI supports TCP, UDP, HTTP(S) in Service definitions and can configure the Octavia listeners with the protocols defined in the Service. OpenStack Octavia supports TCP, UDP and HTTP(S) on listeners, an own listener must be configured for each protocol, and different listeners can be used on the same LB instance. -Summary: the OpenStack based clouds that use Octavia v2 as their LBaaS seems to support this feature once implemented. Pricing is up to their model. +There was a bug in Octavia versions <5.0.0: it was not possible to use the same port number (5e.g. 53) with different protocols (e.g. TCP and UDP) on the same LB instance. It has been fixed in 5.0.0, which is available since the "T" release of OpenStack. + +Summary: the OpenStack based clouds that use Octavia v2 as their LBaaS seems to support this feature once implemented. It is true that the "T" release is the newest one, so upgrade may take a while. On the other hand OpenStack documentation mentions, that a newer Octavia version can be used with previous releases of other OpenStack projects, i.e. it can be the case that the upgrade effort is on Octavia side in an OpenStack cloud. +Pricing is up to the pricing model of the OpenStack providers. #### Oracle Cloud From 1e07f1e1c6ad924d22bd3ab844ec5279d8933eb3 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Thu, 16 Jan 2020 22:31:41 +0100 Subject: [PATCH 04/17] Own findings: - IBM Cloud CPI and LB functionality got more precise description - Oracle CPI and LB functionality got more precise description - Tencent Cloud CPI and LB functionality got more precise description Review comments: - Analyze problems with API downgrade/upgrade if this change is implemented - Change "hardcoded limitation" to "API validation" --- .../sig-network/20200103-mixed-protocol-lb.md | 50 ++++++++++++++++--- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index e7538beac2d..d31d85e090d 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -95,7 +95,7 @@ The goals of this KEP are: ## Proposal -The first thing proposed here is to lift the hardcoded limitation in Kubernetes that currently rejects Service definitions with different protocols if their type is LoadBalancer. Kubernetes would not reject Service definitions like this from that point: +The first thing proposed here is to relax the API validation in Kubernetes that currently rejects Service definitions with different protocols if their type is LoadBalancer. Kubernetes would not reject Service definitions like this from that point: ```yaml apiVersion: v1 kind: Service @@ -135,7 +135,7 @@ https://www.alibabacloud.com/help/doc-detail/74809.htm A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"`. Internal SLBs are free. -Summary: Alibaba CPI and SLB seems to support mixed protocol Services, and the pricing is not based on the number of protocols per Service. +Summary: Alibaba CPI and SLB seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service. #### AWS @@ -203,7 +203,11 @@ That is, if we consider the "single Service with 6 ports" case the user has to p #### IBM Cloud -The IBM Cloud CPI implementation supports TCP, UDP, HTTP protocol values in K8s Services, and it supports mutiple protocols in a Service. The IBM Cloud CPI creates VPC Load Balancer in a VPC based cluster and NLB in a classic cloud based cluster. +The IBM Cloud creates VPC Load Balancer in a VPC based cluster and NLB in a classic cloud based cluster. + +The IBM Cloud CPI implementation for the classic cluster supports TCP and UDP protocol values in K8s Services, and it supports different protocol values in a Service. + +The IBM Cloud CPI implementation for the VPC clusters supports only TCP. The VPC Load Balancer supports TCP and HTTP, and it is possible to create TCP and HTTP listeners for the same LB instance. UDP is not supported. @@ -211,7 +215,7 @@ The VPC LB pricing is time and data amount based, i.e. the number of protocols o The NLB supports TCP and UDP on the same NLB instance. The usage of NLB does not have pricing effects, it is part of the IKS basic package. -Summary: once this feature is implemented IBM Cloud VPC LB can use TCP and HTTP ports from a single Service definition. NLB can use TCP and UDP ports from a single Service definition. +Summary: once this feature is implemented in the K8s API Server the IBM Cloud VPC LB can still use only TCP ports from a Service definition. NLB can use TCP and UDP ports from a single Service definition. #### OpenStack @@ -225,22 +229,31 @@ Pricing is up to the pricing model of the OpenStack providers. #### Oracle Cloud -Oracle Cloud supports TCP, HTTP(S) protocols in its LB solution. The Oracle CPI also supports the protocols in he K8s Service definitions. +The Oracle CPI does not support UDP in the K8s Service definition. It supports only TCP and HTTP. It supports mixed TCP and HTTP ports in the same Service definition. + +The Oracle Cloud Load Balancer supports TCP, HTTP(S) protocols. The pricing is based on time and capacity: https://www.oracle.com/cloud/networking/load-balancing-pricing.html I.e. the amount of protocols on a sinlge LB instance does not affect the pricing. +Summary: Oracle CPI and LB seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service. + #### Tencent Cloud -The Tencent Cloud CPI supports TCP, UDP and HTTP(S) in Service definitions and can configure the CLB listeners with the protocols defined in the Service. +The Tencent Cloud CPI supports TCP, UDP and HTTP in Service definitions. It maps "HTTP" protocol value in a Service definition to "TCP" before creating a listener on the CLB. +The Tencent Cloud CPI can manage both of their LB solutions: "Classic CLB" and "Cloud Load Balancer" (previously known as "Application Load Balancer"). The Tencent Cloud CLB supports TCP, UDP and HTTP(S) listeners, an own listener must be configured for each protocol. The number of listeners does not affect CLB pricing. CLB pricing is time (day) based and not tied to the number of listeners. https://intl.cloud.tencent.com/document/product/214/8848 A user can ask for an internal Load Balancer via a K8s Service definition that has the annotation `service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: subnet-xxxxxxxx`. Internal CLBs are free. +Summary: Tencent Cloud CPI and LBs seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service. + ### Risks and Mitigations +#### Billing perspective + The goal of the current restriction on the K8s API was to prevent an unexpected extra charging for Load Balancers that were created based on Services with mixed protocol definitions. If the current limitation is removed without any option control we introduce the same risk. Let's see which clouds are exposed: - Alibaba: the pricing here is not protocol or forwarding rule or listener based. No risk. @@ -252,6 +265,30 @@ If the current limitation is removed without any option control we introduce the - Oracle: no risk - Tencent Cloud: no risk +#### API change and upgrade/downgrade situations + +In we relax here a validation rule, which is considered as an API-breaking act by the [K8s API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) Even if the change is implemented behind a feature flag the following actions can cause problems: +- the user creates a Service with mixed protocols and then + - turns off the feature flag; or + - executes K8s version rollback to the N-1 version where this feature is not available at all + +When investigating the possible issues with such a change we must consider [the supported version skew among components](https://deploy-preview-11060--kubernetes-io-master-staging.netlify.com/docs/setup/version-skew-policy/), which are the K8s API server and the cloud controller managers (CPI implementations) in our case. + +First of all, feature gate based (a.k.a conditional) field validation must be implemented as defined in the [API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version) One can see a good example for a case when an existing field got a new optional value [here](https://github.com/kubernetes/kubernetes/issues/72651). This practice ensures that upgrade between _future releases_ is safe. Also it enables the further management of existing API objects that were created before turning off the feature flag. Though it does not save us when a K8s API server version rollback is executed to a release in which this feature is not implemented at all. + +Our feature does not introduce new values or new fields. It enables the usage of an existing value in existing fields, but with a different logic. I.e. if someone creates a Service with mixed protocol setup and then rollbacks the API server to a version that does not implement this feature the clients will still get the Service with mixed protocols when they read that via the rollback'd API. If the client (CPI implementation) has been rollbacked, too, then the client may receive such a Service setup that it does not support. + +- Alibaba: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. If this feature is enabled in an API server and then the API server rollback is executed the CPI can still handle the Services with mixed protocol sets. +- AWS: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues. +- Azure: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. The situation is the same as with the Alibaba CPI. +- GCE: currently the GCE CPI assumes that a Service definition contains a single protocol value, as it assumes that the Service Controller already rejected Services with mixed protocols. While the Service Controller really did so a while ago, it does not do this anymore. It means a risk. +- IBM Cloud VPC: no risk. The same situation like in the case of AWS. +- IBM Cloud Classic: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba. +- OpenStack: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba. +- Oracle: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba. +- Tencent Cloud: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba. + + The other risk is in the introduction of this feature without an option control mechanism, i.e. as a general change in Service handling. In that case there is the question whether this feature should be a part of the conformance test set, because it can affect the conformance of cloud providers. A possible mitigation is to put the feature behind option control. @@ -284,6 +321,7 @@ The same works in GCE as a workaround. I.e. if a cloud provider wants to support this feature the CPI must have a logic to apply the Service definitions with a common key value (for example loadBalancerIP) on the same LoadBalancer instance. If the CPI does not implement this support it will work as it does currently. Pro: - the cloud provider can decide when to support this feature, and until that it works as currently for this kind of config +- the restriction on mixed protocols can remain in the K8s API - i.e. there is no K8s API change, and the corresponding risks are mitigated Con: - the users must maintain two Service instances - Atomic update can be a problem - the key must be such a Service attribute that cannot be patched From abb51eb461fb59d3c9c24bae7ff0b47ef6f82157 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Fri, 17 Jan 2020 10:18:50 +0100 Subject: [PATCH 05/17] Fix review findings: - fix the description of the forwarding rule management and billing for different ports with the same protocol Updates: - Test plan added - Graduation criteria added - Downgrade strategy added - Version skew strategy added --- .../sig-network/20200103-mixed-protocol-lb.md | 63 +++++++------------ 1 file changed, 24 insertions(+), 39 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index d31d85e090d..9517bdea01a 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -114,7 +114,7 @@ spec: app: my-dns-server ``` -Once that limit is removed those Service definitions will reach the Cloud Provider LB controller implementations. The logic of the particular Cloud Provider LB controller and of course the actual capabilities and architecture of the backing Cloud Load Balancer services determines how the actual exposure of the application really manifests. For this reason it is important to understand the capabilities of those backing services and to design this feature accordingly. +Once that limit is removed those Service definitions will reach the Cloud Provider LB controller implementations. The logic of the particular Cloud Provider LB controller and of course the actual capabilities and architecture of the backing Cloud Load Balancer services determine how the actual exposure of the application really manifests. For this reason it is important to understand the capabilities of those backing services and to design this feature accordingly. ### User Stories @@ -188,8 +188,6 @@ HTTP protocol support: there is a different LB type in GCE for HTTP traffic: HTT Forwarding rule pricing is per rule: there is a flat price up to 5 forwarding rule instances, on top of which every new forwarding rule has additional cost. -https://cloud.google.com/compute/network-pricing#forwarding_rules_charges - [GCP forwarding_rules_charges](https://cloud.google.com/compute/network-pricing#forwarding_rules_charges) suggest that the same K8s Service definition would result in the creation of 2 forwarding rules in GCP. This has the same fixed price up to 5 forwarding rule instances, and each additional rule results in extra cost. A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `cloud.google.com/load-balancer-type: "Internal"`. Forwarding rules are also part of the GCE Internal TCP/UDP Load Balancer architecture, but in case of Internal TCP/UDP Load Balancer it is not supported to define different forwarding rules with different protocols for the same IP address. That is, for Services with type=LoadBalancer and with annotation `cloud.google.com/load-balancer-type: "Internal"` this feature would not be supported. @@ -197,9 +195,8 @@ A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definitio Summary: The implementation of this feature can affect the bills of GCP users. However the following perspectives are also observed: - if a user wants to have UDP and TCP ports behind the same NLB two Services with must be defined, one for TCP and one for UDP. As the pricing is based on the number of forwarding rules this setup also means the same pricing as with the single Service instance. - if a user is happy with 2 NLB (Service) instances for TCP and UDP still the user has two more forwarding rules to be billed - i.e. it has the same effect on pricing as if those TCP and UDP endpoints were behind the same NLB instance -- already now the bills of users is affected if they have more than 5 forwarding rules as the result of their current Service definitions (e.g. 6 or more port definitions in a single Serice, or if the number of all port definitions in different Services is 6 or more, etc.) -That is, if we consider the "single Service with 6 ports" case the user has to pay more for that single Service ("single LB instance") than for another Service (another LB instance) with 5 or less ports already now. It is not the number of LBs that matters. This phenomenon is already there with the current practice, and the enabling of mixed protocols will not change it to the worse. +That is, if a user wants to use different protocols on the same LB that can be achieved with 2 Service definitions with the current GCP services now. It is not the number of LBs or Service definitions that matters. This phenomenon is already there with the current practice, and the enabling of mixed protocols will not change it to the worse. #### IBM Cloud @@ -259,7 +256,7 @@ If the current limitation is removed without any option control we introduce the - Alibaba: the pricing here is not protocol or forwarding rule or listener based. No risk. - AWS: there is no immediate impact on the pricing side as the AWS CPI limits the scope of protocols to TCP only. - Azure: Azure pricing is indeed based on load-balancing rules, but this cloud already supports mixed protocols via annotations. There is another risk for Azure, though: if the current restriction is removed from the K8s API, the Azure CPI must be prepared to handle Services with mixed protocols. -- GCE: here the risk is valid once the user exceeds the threshold of 5 forwarding rules. Though, as we mentioned above, it is already possible now without this feature +- GCE: here the risk is valid once the user exceeds the threshold of 5 forwarding rules. - IBM Cloud: no risk - OpenStack: here the risk is that there is almost no chance to analyze all the public OpenStack cloud providers with regard to their pricing policies - Oracle: no risk @@ -267,14 +264,14 @@ If the current limitation is removed without any option control we introduce the #### API change and upgrade/downgrade situations -In we relax here a validation rule, which is considered as an API-breaking act by the [K8s API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) Even if the change is implemented behind a feature flag the following actions can cause problems: +We relax here a validation rule, which is considered as an API-breaking act by the [K8s API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) Even if the change is implemented behind a feature flag the following actions can cause problems: - the user creates a Service with mixed protocols and then - turns off the feature flag; or - executes K8s version rollback to the N-1 version where this feature is not available at all When investigating the possible issues with such a change we must consider [the supported version skew among components](https://deploy-preview-11060--kubernetes-io-master-staging.netlify.com/docs/setup/version-skew-policy/), which are the K8s API server and the cloud controller managers (CPI implementations) in our case. -First of all, feature gate based (a.k.a conditional) field validation must be implemented as defined in the [API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version) One can see a good example for a case when an existing field got a new optional value [here](https://github.com/kubernetes/kubernetes/issues/72651). This practice ensures that upgrade between _future releases_ is safe. Also it enables the further management of existing API objects that were created before turning off the feature flag. Though it does not save us when a K8s API server version rollback is executed to a release in which this feature is not implemented at all. +First of all, feature gate based (a.k.a conditional) field validation must be implemented as defined in the [API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version) One can see a good example for a case when an existing field got a new optional value [here](https://github.com/kubernetes/kubernetes/issues/72651). This practice ensures that upgrade and rollback between _future releases_ is safe. Also it enables the further management of existing API objects that were created before turning off the feature flag. Though it does not save us when a K8s API server version rollback is executed to a release in which this feature is not implemented at all. Our feature does not introduce new values or new fields. It enables the usage of an existing value in existing fields, but with a different logic. I.e. if someone creates a Service with mixed protocol setup and then rollbacks the API server to a version that does not implement this feature the clients will still get the Service with mixed protocols when they read that via the rollback'd API. If the client (CPI implementation) has been rollbacked, too, then the client may receive such a Service setup that it does not support. @@ -326,52 +323,40 @@ Con: - the users must maintain two Service instances - Atomic update can be a problem - the key must be such a Service attribute that cannot be patched -### Test Plan - -### Graduation Criteria - -#### Examples +#### Proposed solution -##### Alpha -> Beta Graduation +In the first release: + - a feature flag shall control whether loadbalancer Services with mixed protcol configuration are accepted or not at validation time + - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation + - the update of such services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible + - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still reject such a Service if it cannot manage the corresponding cloud load balancers for those Services. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For exempla, if the Service is not accepted by the CPI implementation this fact shall be available for the user. -- Gather feedback from developers and surveys -- Complete features A, B, C -- Tests are in Testgrid and linked in KEP +In the second release: +- the feature flag shall be set to ON by default. Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. -##### Beta -> GA Graduation +### Test Plan -- N examples of real world usage -- N installs -- More rigorous forms of testing e.g., downgrade tests and scalability tests -- Allowing time for feedback +There must be e2e cases that test whether CPI implementations handle Service definitions with mixed protocol configuration on a consistent way. I.e. either the cloud LB is set up properly or the Service is rejected by the CPI implementation. -##### Removing a deprecated flag +### Graduation Criteria -- Announce deprecation and support policy of the existing flag -- Two versions passed since introducing the functionality which deprecates the flag (to address version skew) -- Address feedback on usage/changed behavior, provided on GitHub issues -- Deprecate the flag +This feature does not need alpha phase. It uses a new feature flag, but from the perspective of graduation it does not need the full transformation path. In the first release the feature flag is turned OFF by default. In the next release it can be turned ON by default. Cloud providers with managed K8s products can still decide when they activate it for the managed clusters. -**For non-optional features moving to GA, the graduation criteria must include [conformance tests].** +### Upgrade / Downgrade Strategy -[conformance tests]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md +#### Downgrade Strategy -### Upgrade / Downgrade Strategy +If a user creates loadbalancer Services with mixed protocols and then the user downgrades to an API Server version that does not support this feature at all the only operation the user can execute on those Services is the delete operation. -If applicable, how will the component be upgraded and downgraded? Make sure this is in the test plan. +If the downgrade also affects the version of the CPI implementation (i.e. when a K8s API server rollback implicitly executes a CPI implementation rollback, too) then the new CPI implementation version may not handle the existing cloud load balancers on a consistent way. It is recommended that the user deletes such Services before the rollback is started to such K8s API server/CPI implementation that does not support this feature. -Consider the following in developing an upgrade/downgrade strategy for this enhancement: -- What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade in order to keep previous behavior? -- What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade in order to make use of the enhancement? +The same stands for the case when the user wants to move the existing Service definitions to another K8s cluster: the user shall check whether the target K8s cluster supports this feature or not and modify the Service descriptors accordingly. ### Version Skew Strategy -If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure -this is in the test plan. +Version skew is possible among the following components in this case: K8s API server, CPI implementation, cloud load balancer -Consider the following in developing a version skew strategy for this enhancement: -- Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used? -- Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet. +Once this feature is implemented in the API server there is a chance that the CPI implementation has to deal with load balancer Services with mixed protocol configuration anytime, even if the API server is downgraded later. The CPI implementation shall be prepared for this, i.e. the CPI implementation cannot expect anymore that the API Server (or any other component, like the Service Controller) filters out such Service definitions. If the CPI implementation wants to have such filtering it has to implement that on its own. In this case the CPI implementation shall be upgraded before the feature is enabled on the API server. In case of a rollback of the API Server such Services can still exist in the cluster, so the CPI implementation should not be downgraded to a version that does not implement that filtering. This is the reason why the CPI implementations shall be updated (if necessary) to be able to deal with such Service definitions already in the first release of this feature. ## Implementation History From 5ca12fd6579ab1a927ee5b82ce5f8c7cd5bb869b Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Sat, 18 Jan 2020 18:27:55 +0100 Subject: [PATCH 06/17] Update based on review comments: - update proposed solution with a better phasing approach throughout the upcoming releases - update graduation criteria --- keps/sig-network/20200103-mixed-protocol-lb.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 9517bdea01a..c11088cb1a8 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -329,10 +329,10 @@ In the first release: - a feature flag shall control whether loadbalancer Services with mixed protcol configuration are accepted or not at validation time - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation - the update of such services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible - - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still reject such a Service if it cannot manage the corresponding cloud load balancers for those Services. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For exempla, if the Service is not accepted by the CPI implementation this fact shall be available for the user. + - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still reject such a Service if it cannot manage the corresponding cloud load balancers for those Services, or for any other reasons. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For exempla, if the Service is not accepted by the CPI implementation this fact shall be available for the user. In the second release: -- the feature flag shall be set to ON by default. Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. +- the feature flag shall be set to ON by default (promoted to beta). Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. ### Test Plan @@ -340,7 +340,12 @@ There must be e2e cases that test whether CPI implementations handle Service def ### Graduation Criteria -This feature does not need alpha phase. It uses a new feature flag, but from the perspective of graduation it does not need the full transformation path. In the first release the feature flag is turned OFF by default. In the next release it can be turned ON by default. Cloud providers with managed K8s products can still decide when they activate it for the managed clusters. +From end user's perspective the graduation criteria are the feecback/bug correction and testing based. + +From CPI implementation perspective thet feature can be graduated to beta, as the cloud providers with managed K8s products can still decide whether they activate it for their managed clusters or not, depending on the status of their CPI implementation. + +Graduating to GA means, that the feature flag checking is removed from the code. It means, that all CPI implementations must be ready to deal with Services with mixed protocol configuration - either rejecting such Services properly or managing the cloud load balancers according to the Service definition. + ### Upgrade / Downgrade Strategy From c7077db9bf5251295e5fccc1a688355b6403e2a4 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Tue, 21 Jan 2020 22:00:52 +0100 Subject: [PATCH 07/17] Fix review comments: - add an example, that the CPI could use a new field in Service.status.loadBalancer or a new Event to indicate that the Service object is not acceptable for it for any reasons (e.g. mixed protocols in the Service) - indicate that the feature flag controls the creation of new Services with mixed protocol config, but not the update operations --- keps/sig-network/20200103-mixed-protocol-lb.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index c11088cb1a8..b9ec31e48eb 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -326,10 +326,10 @@ Con: #### Proposed solution In the first release: - - a feature flag shall control whether loadbalancer Services with mixed protcol configuration are accepted or not at validation time + - a feature flag shall control whether new loadbalancer Services with mixed protcol configuration can be created or not - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation - - the update of such services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible - - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still reject such a Service if it cannot manage the corresponding cloud load balancers for those Services, or for any other reasons. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For exempla, if the Service is not accepted by the CPI implementation this fact shall be available for the user. + - the update of such Services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible + - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still validate the Service objects to check whether any of those has mixed protocol configuration and if the CPI cannot support such Service for any reasons the CPI shall indicate that clearly to the user, and the CPI should not create Cloud LB resources for the "accepted part" of the Service. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For example, if the Service is not accepted by the CPI implementation this fact shall be available for the user, either as a new field in `Service.status.loadBalancer` or as a new Event (or both). In the second release: - the feature flag shall be set to ON by default (promoted to beta). Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. From 9ae1035eff133cd69d20fc9c4882962ad2610fcd Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Thu, 6 Feb 2020 23:27:47 +0100 Subject: [PATCH 08/17] Update ToC and last updated date --- keps/sig-network/20200103-mixed-protocol-lb.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index b9ec31e48eb..b40776a0fbc 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -13,7 +13,7 @@ approvers: - "@thockin" editor: TBD creation-date: 2020-01-03 -last-updated: 2020-01-03 +last-updated: 2020-02-06 status: provisional see-also: replaces: @@ -31,9 +31,9 @@ superseded-by: - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) - - [User Stories [optional]](#user-stories-optional) + - [User Stories](#user-stories) - [Story 1](#story-1) - - [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional) + - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) - [Alibaba](#alibaba) - [AWS](#aws) - [Azure](#azure) @@ -43,17 +43,17 @@ superseded-by: - [Oracle Cloud](#oracle-cloud) - [Tencent Cloud](#tencent-cloud) - [Risks and Mitigations](#risks-and-mitigations) + - [Billing perspective](#billing-perspective) + - [API change and upgrade/downgrade situations](#api-change-and-upgradedowngrade-situations) - [Design Details](#design-details) - [Option Control Alternatives](#option-control-alternatives) - - [Annotiation in the Service definition](#annotiation-in-the-service-definition) + - [Annotation in the Service definition](#annotation-in-the-service-definition) - [Merging Services in CPI](#merging-services-in-cpi) + - [Proposed solution](#proposed-solution) - [Test Plan](#test-plan) - [Graduation Criteria](#graduation-criteria) - - [Examples](#examples) - - [Alpha -> Beta Graduation](#alpha---beta-graduation) - - [Beta -> GA Graduation](#beta---ga-graduation) - - [Removing a deprecated flag](#removing-a-deprecated-flag) - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Downgrade Strategy](#downgrade-strategy) - [Version Skew Strategy](#version-skew-strategy) - [Implementation History](#implementation-history) - [Drawbacks [optional]](#drawbacks-optional) From c54b92d9446f36ebbe01a6623aff20b35db716fe Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Mon, 9 Mar 2020 21:46:50 +0100 Subject: [PATCH 09/17] Fix review comments. Add DigitalOcean part --- keps/sig-network/20200103-mixed-protocol-lb.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index b40776a0fbc..1a7e7617787 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -33,10 +33,12 @@ superseded-by: - [Proposal](#proposal) - [User Stories](#user-stories) - [Story 1](#story-1) + - [Story 2](#story-2) - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) - [Alibaba](#alibaba) - [AWS](#aws) - [Azure](#azure) + - [DigitalOcean](#digitalocean) - [GCE](#gce) - [IBM Cloud](#ibm-cloud) - [OpenStack](#openstack) @@ -120,7 +122,11 @@ Once that limit is removed those Service definitions will reach the Cloud Provid #### Story 1 -As a Kubernetes cluster user I want to expose an application that provides its service on different protocols with a single cloud provider load balancer IP. In order to achieve this I want to define different `ports` with mixed protocols in my Service definition of `type: LoadBalancer` +As a Kubernetes cluster user I want to have the capability of exposing a DNS server for TCP and UDP based requests on the same Load Balancer IP address. See [RFC7766](https://tools.ietf.org/html/rfc7766). In order to achieve this I want to define different `ports` with mixed protocols in my Service definition of `type: LoadBalancer` + +#### Story 2 + +As as Kubernetes cluster user I want to have the capability of exposing my SIP Server for TCP and UDP based requests on the same Load Balacer IP address and port. This requirement comes from [RFC3261 Section 18.2.1](https://tools.ietf.org/html/rfc3261#section-18.2.1) ### Implementation Details/Notes/Constraints @@ -174,6 +180,12 @@ A user can ask for an internal Azure Load Balancer via a K8s Service definition Summary: Azure already has mixed protocol support for TCP and UDP, and it already affects the bills of the users. The implementation of this feature may require some work in Azure CPI. +#### DigitalOcean + +The DigitalOcean CPI does not support mixed protocols in the Service definition, it accepts TCP only for load balancer Services. It is possible to ask for HTTP(S) and HTTP2 ports with Service annotations. + +Summary: The DO CPI is the current bottleneck with its TCP-only limitation. As long as it is there the implementation of this feature will have no effect on the DO bills. + #### GCE The GCE CPI supports only TCP and UDP protocols in Services. @@ -269,7 +281,7 @@ We relax here a validation rule, which is considered as an API-breaking act by t - turns off the feature flag; or - executes K8s version rollback to the N-1 version where this feature is not available at all -When investigating the possible issues with such a change we must consider [the supported version skew among components](https://deploy-preview-11060--kubernetes-io-master-staging.netlify.com/docs/setup/version-skew-policy/), which are the K8s API server and the cloud controller managers (CPI implementations) in our case. +When investigating the possible issues with such a change we must consider [the supported version skew among components](https://kubernetes.io/docs/setup/release/version-skew-policy/), which are the K8s API server and the cloud controller managers (CPI implementations) in our case. First of all, feature gate based (a.k.a conditional) field validation must be implemented as defined in the [API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version) One can see a good example for a case when an existing field got a new optional value [here](https://github.com/kubernetes/kubernetes/issues/72651). This practice ensures that upgrade and rollback between _future releases_ is safe. Also it enables the further management of existing API objects that were created before turning off the feature flag. Though it does not save us when a K8s API server version rollback is executed to a release in which this feature is not implemented at all. @@ -279,6 +291,7 @@ Our feature does not introduce new values or new fields. It enables the usage of - AWS: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues. - Azure: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. The situation is the same as with the Alibaba CPI. - GCE: currently the GCE CPI assumes that a Service definition contains a single protocol value, as it assumes that the Service Controller already rejected Services with mixed protocols. While the Service Controller really did so a while ago, it does not do this anymore. It means a risk. +- DigitalOcean: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues. - IBM Cloud VPC: no risk. The same situation like in the case of AWS. - IBM Cloud Classic: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba. - OpenStack: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba. From dfa13387884ca31ccd97d319999bea3e49e726ed Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Sun, 29 Mar 2020 21:33:26 +0200 Subject: [PATCH 10/17] Fix review comments. Describe in Motivation why billing is a central matter of the KEP --- keps/sig-network/20200103-mixed-protocol-lb.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 1a7e7617787..90b3047c2d0 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -13,7 +13,7 @@ approvers: - "@thockin" editor: TBD creation-date: 2020-01-03 -last-updated: 2020-02-06 +last-updated: 2020-03-29 status: provisional see-also: replaces: @@ -85,6 +85,10 @@ The following issue and PR shows considerable interest from users that would ben https://github.com/kubernetes/kubernetes/issues/23880 https://github.com/kubernetes/kubernetes/pull/75831 +The current restriction that rejects a Service creation request if it has two different protocol definitions was introduced because some cloud implementations may charge their load balancer users on a "per protocol" basis. That is, the current logic is to prevent negative surprises with the load balancer bills. The current implementation enforces a more explicit statement or consent form the end user for the usage of two different protocols on the same load balancer. For example GCE expects that the user creates two load balancer Service definitions for the two different protocols. + +But such workarounds or solutions do not exist in all cloud implementations. According to the feedback from the end users it would be more beneficial to remove this restriction from the Kubernetes code. This KEP is to investigate how the removal of that restriction would affect the billing of load balancer resources in the different clouds, i.e. whether it is safe or not to allow the usage of different protocol values in the same Service definition. + ### Goals The goals of this KEP are: From 511feeae848206f4e652cb42c5087f22acc09bcf Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Wed, 8 Jul 2020 18:05:57 +0000 Subject: [PATCH 11/17] Fix review comments. Update AWS information: AWS CPI supports UDP and TCP for AWS NLB. Be more specific about the selected solution, and the requirements agains the CPIs. Propose a schema for the extension of the LoadBalancerStatus field in ServiceStatus to provide information about the ports of the load-balancer.x --- .../sig-network/20200103-mixed-protocol-lb.md | 89 ++++++++++++++++--- 1 file changed, 76 insertions(+), 13 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 90b3047c2d0..51c651f657e 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -149,21 +149,25 @@ Summary: Alibaba CPI and SLB seem to support mixed protocol Services, and the pr #### AWS -The AWS CPI does not support mixed protocols in the Service definition since it only allows TCP for load balancers. The AWS CPI looks for annotations on the Service to determine whether TCP, TLS or HTTP(S) listener should be created in the AWS ELB for a configured Service port. +The AWS CPI supports TCP and UDP protocols in Service definitions when an AWS NLB is requested. However, the AWS CPI cannot set up an AWS NLB with TCP and UDP ports behind the same IP address for the same port number currently (for example, the DNS use case that wants to open port 53 with both UDP and TCP would not work right now). See https://github.com/kubernetes/kubernetes/pull/92109#discussion_r439730341 + +For AWS ELB only TCP is accepted by the CPI. + +The AWS CPI looks for annotations on the Service to determine whether TCP, TLS or HTTP(S) listener should be created in the AWS ELB for a configured Service port. AWS Classic LB supports TCP,TLS, and HTTP(S) protocols behind the same IP address. -AWS Network LB supports TCP/TLS and UDP protocols behind the same IP address. As we can see, UDP cannot be utilized currently, due to the limitation in the AWS CPI. +AWS Network LB supports TCP/TLS and UDP protocols behind the same IP address. The usage of TCP+HTTP or UDP+HTTP on the same LB instace behind the same IP address is not possible in AWS. From a pricing perspective the AWS NLB and the CLB have the following models: https://aws.amazon.com/elasticloadbalancing/pricing/ -Both are primarily usage based rather than charging based on the number of protocol, however NLBs have separate pricing unit quotas for TCP, UDP and TLS. +Both are usage based rather than based on the number of protocols, however NLBs have separate pricing unit quotas for TCP, UDP and TLS. A user can ask for an internal Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0`. So far the author could not find any difference in the usage and pricing of those when compared to the external LBs - except the pre-requisite of a private subnet on which the LB can be deployed. -Summary: AWS CPI is the current bottleneck with its TCP-only limitation. As long as it is there the implementation of this feature will have no effect on the AWS bills. +Summary: AWS CPI supports mixed protocols (TCP and UDP) on AWS NLBs, but the support of the same port number with different protocols still requires some work in the CPI . Once this feature is implemented in the K8s API the users can request for such mixed-protocol NLBs via their Service definitions, and the users will be charged according to the rules defined in the pricing rules. #### Azure @@ -292,7 +296,7 @@ First of all, feature gate based (a.k.a conditional) field validation must be im Our feature does not introduce new values or new fields. It enables the usage of an existing value in existing fields, but with a different logic. I.e. if someone creates a Service with mixed protocol setup and then rollbacks the API server to a version that does not implement this feature the clients will still get the Service with mixed protocols when they read that via the rollback'd API. If the client (CPI implementation) has been rollbacked, too, then the client may receive such a Service setup that it does not support. - Alibaba: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. If this feature is enabled in an API server and then the API server rollback is executed the CPI can still handle the Services with mixed protocol sets. -- AWS: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues. +- AWS: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. The situation is the same as with the Alibaba CPI. - Azure: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. The situation is the same as with the Alibaba CPI. - GCE: currently the GCE CPI assumes that a Service definition contains a single protocol value, as it assumes that the Service Controller already rejected Services with mixed protocols. While the Service Controller really did so a while ago, it does not do this anymore. It means a risk. - DigitalOcean: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues. @@ -302,19 +306,16 @@ Our feature does not introduce new values or new fields. It enables the usage of - Oracle: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba. - Tencent Cloud: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba. - -The other risk is in the introduction of this feature without an option control mechanism, i.e. as a general change in Service handling. In that case there is the question whether this feature should be a part of the conformance test set, because it can affect the conformance of cloud providers. - -A possible mitigation is to put the feature behind option control. +As stated above we must implement a feature gate based phased introduction for this feature because of its effects. See the `Proposed solution` part for details in this document below. ## Design Details The implementation of the basic logic is ready in this PR: https://github.com/kubernetes/kubernetes/pull/75831 -Currently a feature gate is used to control its activation status. Though if we want to keep this feature behind option control even after it reaches its GA state we should come up with a different solution, as feature gates are used to control the status of a feature as that graduates from alpha to GA, and they are not meant for option control for features with GA status. +The current implementation has a feature gate to control its activation status. -### Option Control Alternatives +### Option Control Alternatives - considered alternatives #### Annotation in the Service definition @@ -340,17 +341,79 @@ Con: - the users must maintain two Service instances - Atomic update can be a problem - the key must be such a Service attribute that cannot be patched -#### Proposed solution +### The selected solution for the option control In the first release: - a feature flag shall control whether new loadbalancer Services with mixed protcol configuration can be created or not - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation - the update of such Services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible - - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still validate the Service objects to check whether any of those has mixed protocol configuration and if the CPI cannot support such Service for any reasons the CPI shall indicate that clearly to the user, and the CPI should not create Cloud LB resources for the "accepted part" of the Service. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For example, if the Service is not accepted by the CPI implementation this fact shall be available for the user, either as a new field in `Service.status.loadBalancer` or as a new Event (or both). + - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. Either via supporting such Service definitions, or clearly indicating to the users that the Service could not be processed as specified. As we can see from our analysis some CPIs support other protocols than TCP and UPD in the Service definitions, while others support only TCP and UDP. That is the term "mixed protocol support" does not always mean that all possible protocol values are supported by a CPI. For this reason a nicely behaving CPI shall + - indicate clearly to the user what ports with what protocols have been opened on the LB + - preferably not create any Cloud LB resources if the Service definition contains unsupported protocols. + + In order to provide a way for the CPI to indicate port statuses to the user we would add the following new `portStatus` list of port statuses to `Service.status.loadBalancer`, so the CPI can indicate the status of the LB: + +``` +"io.k8s.api.core.v1.PortStatus": { + "description": "PortStatus contains details for the current status of this port.", + "properties": { + "port": { + "description": "Port number", + "format": "int32", + "type": "integer" + }, + "protocol": { + "description": "The protocol for this port", + "type": "string" + }, + "name": { + "description": "The name of this port within the service.", + "type": "string" + }, + "ready": { + "description": "Specifies whether the port was configured for the load-balancer.", + "type": "boolean" + }, + "message": { + "description": "A human readable message indicating details about why the port is in this condition.", + "type": "string" + } + }, + "type": "object" +}, +"io.k8s.api.core.v1.LoadBalancerStatus": { + "description": "LoadBalancerStatus represents the status of a load-balancer.", + "properties": { + "ingress": { + "description": "Ingress is a list containing ingress points for the load-balancer. Traffic intended for the service ld be sent to these ingress points.", + "items": { + "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerIngress" + }, + "type": "array" + }, + "portStatuses": { + "description": "The list has one entry per port in the manifest. + "items": { + "$ref": "#/definitions/io.k8s.api.core.v1.PortStatus" + }, + "type": "array" + }, + "message": { + "description": "A human readable message indicating details about the condition of the load-balancer.", + "type": "string" + } + }, + "type": "object" +}, + +A CPI shall also set an Event in case it cannot create a Cloud LB instance that could fulfill the Service specification. In the second release: - the feature flag shall be set to ON by default (promoted to beta). Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. +In the lomg term: +- the feature flag is removed and the feature becomes generic without option control + ### Test Plan There must be e2e cases that test whether CPI implementations handle Service definitions with mixed protocol configuration on a consistent way. I.e. either the cloud LB is set up properly or the Service is rejected by the CPI implementation. From 7fee833ee444a516e528c931153c71102a39bc0c Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Wed, 8 Jul 2020 18:13:10 +0000 Subject: [PATCH 12/17] Terminate code block for the schema --- keps/sig-network/20200103-mixed-protocol-lb.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 51c651f657e..d3645758dd2 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -353,7 +353,7 @@ In the first release: In order to provide a way for the CPI to indicate port statuses to the user we would add the following new `portStatus` list of port statuses to `Service.status.loadBalancer`, so the CPI can indicate the status of the LB: -``` +```json "io.k8s.api.core.v1.PortStatus": { "description": "PortStatus contains details for the current status of this port.", "properties": { @@ -405,6 +405,7 @@ In the first release: }, "type": "object" }, +``` A CPI shall also set an Event in case it cannot create a Cloud LB instance that could fulfill the Service specification. From 8fe696e89c487dfa4816a7a07f93bd9d18e1204a Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Wed, 8 Jul 2020 18:20:54 +0000 Subject: [PATCH 13/17] Update TOC --- keps/sig-network/20200103-mixed-protocol-lb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index d3645758dd2..d3143ad6ecf 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -48,10 +48,10 @@ superseded-by: - [Billing perspective](#billing-perspective) - [API change and upgrade/downgrade situations](#api-change-and-upgradedowngrade-situations) - [Design Details](#design-details) - - [Option Control Alternatives](#option-control-alternatives) + - [Option Control Alternatives - considered alternatives](#option-control-alternatives---considered-alternatives) - [Annotation in the Service definition](#annotation-in-the-service-definition) - [Merging Services in CPI](#merging-services-in-cpi) - - [Proposed solution](#proposed-solution) + - [The selected solution for the option control](#the-selected-solution-for-the-option-control) - [Test Plan](#test-plan) - [Graduation Criteria](#graduation-criteria) - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) From 2416c12c5ab6caf9e7ac1f68a39d2a296cd860e0 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Thu, 9 Jul 2020 17:29:40 +0000 Subject: [PATCH 14/17] Fix UPD to UDP. Fix last-updated --- keps/sig-network/20200103-mixed-protocol-lb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index d3143ad6ecf..6b83ba2b68d 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -13,7 +13,7 @@ approvers: - "@thockin" editor: TBD creation-date: 2020-01-03 -last-updated: 2020-03-29 +last-updated: 2020-07-09 status: provisional see-also: replaces: @@ -347,7 +347,7 @@ In the first release: - a feature flag shall control whether new loadbalancer Services with mixed protcol configuration can be created or not - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation - the update of such Services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible - - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. Either via supporting such Service definitions, or clearly indicating to the users that the Service could not be processed as specified. As we can see from our analysis some CPIs support other protocols than TCP and UPD in the Service definitions, while others support only TCP and UDP. That is the term "mixed protocol support" does not always mean that all possible protocol values are supported by a CPI. For this reason a nicely behaving CPI shall + - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. Either via supporting such Service definitions, or clearly indicating to the users that the Service could not be processed as specified. As we can see from our analysis some CPIs support other protocols than TCP and UDP in the Service definitions, while others support only TCP and UDP. That is the term "mixed protocol support" does not always mean that all possible protocol values are supported by a CPI. For this reason a nicely behaving CPI shall - indicate clearly to the user what ports with what protocols have been opened on the LB - preferably not create any Cloud LB resources if the Service definition contains unsupported protocols. From de228b43206e6b4f3bb0401b5f8cf2631325415b Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Sat, 11 Jul 2020 20:32:26 +0000 Subject: [PATCH 15/17] GCE internal LB supports different protocols in the forwarding rules. Move portStatus inside the ingress definition of an LB. Add a conditions field to the Service.status.loadBalancer and add the first official condition "LoadBalancerMixedProtocolNotSupported to it. --- .../sig-network/20200103-mixed-protocol-lb.md | 75 +++++++++++++++---- 1 file changed, 60 insertions(+), 15 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 6b83ba2b68d..643b9d7a1d2 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -210,7 +210,7 @@ Forwarding rule pricing is per rule: there is a flat price up to 5 forwarding ru [GCP forwarding_rules_charges](https://cloud.google.com/compute/network-pricing#forwarding_rules_charges) suggest that the same K8s Service definition would result in the creation of 2 forwarding rules in GCP. This has the same fixed price up to 5 forwarding rule instances, and each additional rule results in extra cost. -A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `cloud.google.com/load-balancer-type: "Internal"`. Forwarding rules are also part of the GCE Internal TCP/UDP Load Balancer architecture, but in case of Internal TCP/UDP Load Balancer it is not supported to define different forwarding rules with different protocols for the same IP address. That is, for Services with type=LoadBalancer and with annotation `cloud.google.com/load-balancer-type: "Internal"` this feature would not be supported. +A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `cloud.google.com/load-balancer-type: "Internal"`. Forwarding rules are part of the GCE Internal TCP/UDP Load Balancer architecture, too, and the user can define different forwarding rules with different protocols for the same IP address. Summary: The implementation of this feature can affect the bills of GCP users. However the following perspectives are also observed: - if a user wants to have UDP and TCP ports behind the same NLB two Services with must be defined, one for TCP and one for UDP. As the pricing is based on the number of forwarding rules this setup also means the same pricing as with the single Service instance. @@ -351,9 +351,35 @@ In the first release: - indicate clearly to the user what ports with what protocols have been opened on the LB - preferably not create any Cloud LB resources if the Service definition contains unsupported protocols. - In order to provide a way for the CPI to indicate port statuses to the user we would add the following new `portStatus` list of port statuses to `Service.status.loadBalancer`, so the CPI can indicate the status of the LB: + In order to provide a way for the CPI to indicate port statuses to the user we would add the following new `portStatus` list of port statuses to `Service.status.loadBalancer.ingress`, so the CPI can indicate the status of the LB. Also we add a `conditions` list of conditions to the `Service.status.loadBalancer`, with the first official condition `LoadBalancerMixedProtocolNotSupported` defined. ```json +"io.k8s.api.core.v1.LoadBalancerCondition": { + "description": "LoadBalancerCondition contains details for the current condition of this load-balancer.", + "properties": { + "type": { + "description": "Type is the type of the condition. Known conditions are \"LoadBalancerMixedProtocolNotSupported\".\n\nThe \"LoadBalancerMixedProtocolNotSupported\" condition with \"True\" status means that the cloud provider implementation could not create the requested load-balancer with the specified set of ports because that set contains different protocol values for the ports, and such a configuration is not supported either by the cloud provider or by the load balancer or both.", + "type": "string" + }, + "status": { + "description": "Status is the status of the condition. Can be True, False, Unknown.", + "type": "string" + }, + "message": { + "description": "Human-readable message indicating details about last transition.", + "type": "string" + }, + "reason": { + "description": "Unique, one-word, CamelCase reason for the condition's last transition.", + "type": "string" + } + }, + "required": [ + "type", + "status" + ], + "type": "object" +}, "io.k8s.api.core.v1.PortStatus": { "description": "PortStatus contains details for the current status of this port.", "properties": { @@ -366,10 +392,6 @@ In the first release: "description": "The protocol for this port", "type": "string" }, - "name": { - "description": "The name of this port within the service.", - "type": "string" - }, "ready": { "description": "Specifies whether the port was configured for the load-balancer.", "type": "boolean" @@ -381,18 +403,19 @@ In the first release: }, "type": "object" }, -"io.k8s.api.core.v1.LoadBalancerStatus": { - "description": "LoadBalancerStatus represents the status of a load-balancer.", +"io.k8s.api.core.v1.LoadBalancerIngress": { + "description": "LoadBalancerIngress represents the status of a load-balancer ingress point: traffic intended for the servicshould be sent to an ingress point.", "properties": { - "ingress": { - "description": "Ingress is a list containing ingress points for the load-balancer. Traffic intended for the service ld be sent to these ingress points.", - "items": { - "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerIngress" - }, - "type": "array" + "hostname": { + "description": "Hostname is set for load-balancer ingress points that are DNS based (typically AWS load-balancers)", + "type": "string" + }, + "ip": { + "description": "IP is set for load-balancer ingress points that are IP based (typically GCE or OpenStack load-balancers)", + "type": "string" }, "portStatuses": { - "description": "The list has one entry per port in the manifest. + "description": "The list has one entry per port in the manifest.", "items": { "$ref": "#/definitions/io.k8s.api.core.v1.PortStatus" }, @@ -405,6 +428,28 @@ In the first release: }, "type": "object" }, +"io.k8s.api.core.v1.LoadBalancerStatus": { + "description": "LoadBalancerStatus represents the status of a load-balancer.", + "properties": { + "ingress": { + "description": "Ingress is a list containing ingress points for the load-balancer. Traffic intended for the servi should be sent to these ingress points.", + "items": { + "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerIngress" + }, + "type": "array" + }, + "conditions": { + "description": "Current service state of the load-balancer", + "items": { + "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerCondition" + }, + "type": "array", + "x-kubernetes-patch-merge-key": "type", + "x-kubernetes-patch-strategy": "merge" + } + }, + "type": "object" +} ``` A CPI shall also set an Event in case it cannot create a Cloud LB instance that could fulfill the Service specification. From 7c5d9eb27cc932d805acfc1cbeea4f5a37317769 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Mon, 13 Jul 2020 18:58:05 +0000 Subject: [PATCH 16/17] Move the new conditions from loadbalancer level to the generic Service.Status level --- .../sig-network/20200103-mixed-protocol-lb.md | 20 ++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 643b9d7a1d2..61c76e0b0fd 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -354,8 +354,8 @@ In the first release: In order to provide a way for the CPI to indicate port statuses to the user we would add the following new `portStatus` list of port statuses to `Service.status.loadBalancer.ingress`, so the CPI can indicate the status of the LB. Also we add a `conditions` list of conditions to the `Service.status.loadBalancer`, with the first official condition `LoadBalancerMixedProtocolNotSupported` defined. ```json -"io.k8s.api.core.v1.LoadBalancerCondition": { - "description": "LoadBalancerCondition contains details for the current condition of this load-balancer.", +"io.k8s.api.core.v1.ServiceCondition": { + "description": "ServiceCondition contains details for the current condition of this Service.", "properties": { "type": { "description": "Type is the type of the condition. Known conditions are \"LoadBalancerMixedProtocolNotSupported\".\n\nThe \"LoadBalancerMixedProtocolNotSupported\" condition with \"True\" status means that the cloud provider implementation could not create the requested load-balancer with the specified set of ports because that set contains different protocol values for the ports, and such a configuration is not supported either by the cloud provider or by the load balancer or both.", @@ -437,11 +437,21 @@ In the first release: "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerIngress" }, "type": "array" + } + }, + "type": "object" +}, +"io.k8s.api.core.v1.ServiceStatus": { + "description": "ServiceStatus represents the current status of a service.", + "properties": { + "loadBalancer": { + "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerStatus", + "description": "LoadBalancer contains the current status of the load-balancer, if one is present." }, "conditions": { - "description": "Current service state of the load-balancer", + "description": "Current service state", "items": { - "$ref": "#/definitions/io.k8s.api.core.v1.LoadBalancerCondition" + "$ref": "#/definitions/io.k8s.api.core.v1.ServiceCondition" }, "type": "array", "x-kubernetes-patch-merge-key": "type", @@ -449,7 +459,7 @@ In the first release: } }, "type": "object" -} +}, ``` A CPI shall also set an Event in case it cannot create a Cloud LB instance that could fulfill the Service specification. From 173bc38c742deca371665398e3ca4c23c452bcf4 Mon Sep 17 00:00:00 2001 From: Laszlo Janosi Date: Tue, 14 Jul 2020 18:34:15 +0000 Subject: [PATCH 17/17] Add how kube-proxy should use the new port status information --- keps/sig-network/20200103-mixed-protocol-lb.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/keps/sig-network/20200103-mixed-protocol-lb.md b/keps/sig-network/20200103-mixed-protocol-lb.md index 61c76e0b0fd..a27c8b56151 100644 --- a/keps/sig-network/20200103-mixed-protocol-lb.md +++ b/keps/sig-network/20200103-mixed-protocol-lb.md @@ -52,6 +52,7 @@ superseded-by: - [Annotation in the Service definition](#annotation-in-the-service-definition) - [Merging Services in CPI](#merging-services-in-cpi) - [The selected solution for the option control](#the-selected-solution-for-the-option-control) + - [Kube-proxy](#kube-proxy) - [Test Plan](#test-plan) - [Graduation Criteria](#graduation-criteria) - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) @@ -467,9 +468,13 @@ A CPI shall also set an Event in case it cannot create a Cloud LB instance that In the second release: - the feature flag shall be set to ON by default (promoted to beta). Most probably we want to keep the feature flag so cloud providers can decide whether they enable it or not in their managed K8s services depending their CPI implementations. -In the lomg term: +In the long term: - the feature flag is removed and the feature becomes generic without option control +### Kube-proxy + +The kube-proxy should use the port status information from `Service.status.loadBalancer.ingress` in order not to allow traffic to those ports that could not be opened by the load balancer either. + ### Test Plan There must be e2e cases that test whether CPI implementations handle Service definitions with mixed protocol configuration on a consistent way. I.e. either the cloud LB is set up properly or the Service is rejected by the CPI implementation.