Use GCE load balancer controller with backend buckets

[bowei]

From @omerzach on February 28, 2017 1:20

We're happily using the GCE load balancer controller in production to route traffic to a few different services. We'd like to have some paths point at backend buckets in Google Cloud Storage instead of backend services running in Kubernetes.

Right now if we manually create this backend bucket and then configure the load balancer to point certain paths at it the UrlMap is updates appropriately but almost immediately reverted to its previous setting, presumably because the controller sees it doesn't match the YAML we initially configured the Ingress with.

I have two questions:

1. Is there any immediate workaround where we could continue using the Kubernetes controller but manually modify the UrlMap to work with a backend bucket?
2. Would a pull request to add support for backend buckets in the load balancer controller be something the community is interested in? We're willing to invest engineering effort into this, though none of our team has experience with Go or the Kubernetes codebase so we might need a bit of guidance.

(For some context, we'd like to do something like this: https://cloud.google.com/compute/docs/load-balancing/http/using-http-lb-with-cloud-storage)

Copied from original issue: kubernetes/ingress-nginx#353

[bowei]

From @bprashanth on March 7, 2017 3:39

I don't think there's an immediate work around, as the controller will construct a url map based on your ingress and sync it continuously.

Something that says: serve static content for these paths from a content cache backed by [S3, GCS, memory etc] sounds like a good idea. We should allow GCE L7 x in-memory cache, but for the first cut we might get away with a simple boolean on the HTTPIngressPath (https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/types.go#L685). We'd have to assume GCS and allocates a private bucket if the Ingress is a GCE lb.

Maybe we should fork into another issue that tackles how we enable CDN on Ingress, and figure out the api part first? @nicksardo @thockin

[bowei]

From @ConradIrwin on March 29, 2017 22:16

@nicksardo thanks for picking this up!

I'd also like to be able to proxy certain paths to CloudStorage — let me know if you want a sounding board for design decisions.

[bowei]

From @bbzg on April 30, 2017 8:14

This would be very useful for us. Has there been any progress since March?

[bowei]

From @thockin on May 1, 2017 4:16

As far as I know, nobody is looking at this right now. What I don't want
to do is make Ingress a proxy-API for all of GCLB..

On Sun, Apr 30, 2017 at 1:15 AM, bbzg [email protected] wrote:

This would be very useful for us. Has there been any progress since
March?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
kubernetes/ingress-nginx#353 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNrjEkp-7zCsHBX1TB1bjBwJkD6eks5r1EMFgaJpZM4MN2NU
.

[bowei]

From @ConradIrwin on May 1, 2017 4:50

Tim. The ingress API is very convenient, but I see your argument. Would it make more sense to make it a GCLB "controller" instead?Sent via SuperhumanOn Sun, Apr 30, 2017 at 9:16 PM, Tim Hockin[email protected]wrote:As far as I know, nobody is looking at this right now. What I don't want
to do is make Ingress a proxy-API for all of GCLB..

On Sun, Apr 30, 2017 at 1:15 AM, bbzg [email protected] wrote:

This would be very useful for us. Has there been any progress since
March?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
kubernetes/ingress-nginx#353 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNrjEkp-7zCsHBX1TB1bjBwJkD6eks5r1EMFgaJpZM4MN2NU
.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

[bowei]

From @thockin on May 1, 2017 5:7

I'm not sure. It could go a couple ways.

We could clearly denote where our controller will revert manual changes and
where it won't, so you could make these changes manually (or by other
controller). We could consider additional annotations for this, but it is
a slippery slope. Other ideas?

On Sun, Apr 30, 2017 at 9:50 PM, Conrad Irwin [email protected]
wrote:

Tim. The ingress API is very convenient, but I see your argument. Would it
make more sense to make it a GCLB "controller" instead?Sent via
SuperhumanOn Sun, Apr 30, 2017 at 9:16 PM, Tim Hockin<
[email protected]>wrote:As far as I know, nobody is looking at
this right now. What I don't want
to do is make Ingress a proxy-API for all of GCLB..

On Sun, Apr 30, 2017 at 1:15 AM, bbzg [email protected] wrote:

This would be very useful for us. Has there been any progress since
March?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<kubernetes/ingress-nginx#353 (comment)
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVNrjEkp-
7zCsHBX1TB1bjBwJkD6eks5r1EMFgaJpZM4MN2NU>
.

—You are receiving this because you commented.Reply to this email
directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
kubernetes/ingress-nginx#353 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVH8mIh19FVzwp-Z5jQnCWRFBV1Iqks5r1WSzgaJpZM4MN2NU
.

[bowei]

From @ConradIrwin on May 1, 2017 5:18

I'd be happy with a less magic API between the two — i.e. I could manually configure a load balancer if kubernetes gave me a backend I could point it to (or maybe just an instance-group + port?)Sent via SuperhumanOn Sun, Apr 30, 2017 at 10:07 PM, Tim Hockin[email protected]wrote:I'm not sure. It could go a couple ways.

We could clearly denote where our controller will revert manual changes and
where it won't, so you could make these changes manually (or by other
controller). We could consider additional annotations for this, but it is
a slippery slope. Other ideas?

On Sun, Apr 30, 2017 at 9:50 PM, Conrad Irwin [email protected]
wrote:

Tim. The ingress API is very convenient, but I see your argument. Would it
make more sense to make it a GCLB "controller" instead?Sent via
SuperhumanOn Sun, Apr 30, 2017 at 9:16 PM, Tim Hockin<
[email protected]>wrote:As far as I know, nobody is looking at
this right now. What I don't want
to do is make Ingress a proxy-API for all of GCLB..

On Sun, Apr 30, 2017 at 1:15 AM, bbzg [email protected] wrote:

This would be very useful for us. Has there been any progress since
March?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<kubernetes/ingress-nginx#353 (comment)
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVNrjEkp-
7zCsHBX1TB1bjBwJkD6eks5r1EMFgaJpZM4MN2NU>
.

—You are receiving this because you commented.Reply to this email
directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
kubernetes/ingress-nginx#353 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVH8mIh19FVzwp-Z5jQnCWRFBV1Iqks5r1WSzgaJpZM4MN2NU
.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

[bowei]

From @gcbirzan on July 20, 2017 13:58

As a workaround for this issue, wouldn't not touching rules that kubernetes cannot create (i.e. those not pointing to a backend service, but to a bucket) be okay?

As a side note, we had this working on 1.6.x, but after upgrading it started removing the extra rules in the url map...

[bowei]

From @c3s4r on July 26, 2017 22:24

Any updates on this? Is it scheduled? Is there a timeline? ... Since I want the content served using https, right now the only workaround I can think on is to manually create another load balancer (not using ingress) just for the static content, which I don't like because it adds the cost of an additional load balancer :(

[bowei]

From @lostpebble on July 29, 2017 10:16

Just coming across this now after finishing setting up backend buckets for our system...

This is a major set back for us trying to set up static file routes alongside our server backends. I agree with @gcbirzan that perhaps the load balancer should be updated for the values that Kubernetes can control rather than replaced wholly (and in the process removing GCP-specific rules).

Right now things feel too flaky to rely on backend buckets for static file serving, if when we update the configuration we might lose those pathways and return bad requests.

Huge pity because the CDN and load bearing capabilities the backend buckets could afford us is a major asset to our system.

[bowei]

From @jakobholmelund on September 27, 2017 13:35

Any news on this ?

[cwenzelg]

Would be also really interested in this feature! we could really need that.

[cwenzelg]

Would there be maybe a possibility to use maye a ignore pattern and the possibility to use a existing load balancer? Then there could be existing url_maps maps ignored (if told in the ingress) same as other existing backends for example. This could also solve the CDN Problem. Then you configure Google Cloud Stuff individually without adding everything in kubernetes.

[bartoszhernas]

+1

[dhagge]

+1 this would be very useful. Current workaround is to use a regular old loadbalancer.

[dehypnosis]

+1 waiting this feature

[thunderball1]

+1

[tgjonestx]

+1

[vansters-dev]

+1

[scottefein]

+1

[fstn]

+1

[zvozin]

You can achieve this by configuring the K8 load-balancing manually as opposed to using an NGINX ingress. Assuming your services are deployed NodePort with a static nodePort indicated in the ports object (the tcp:30000-40000 range seems to be where they go by default and is a good rule to follow), you will need to:

• create a load balancer instance
• create a firewall rule that allows traffic from 130.211.0.0/22 and 35.191.0.0/16 to the K8 instance group
• create a backend service against the K8 instance group, pointing to whatever nodePort you specified above
• not forget a health check

For incoming TLS, create a Kubernetes secret of a cert and assign as per https://cloud.google.com/compute/docs/load-balancing/tcp-ssl

[scottefein]

You could also manage your own ports with the NodePort service to get around the ephemeral port allocation.

…

On Mon, Jan 22, 2018, 9:20 PM Alex Zuzin ***@***.***> wrote: You can achieve this by configuring the K8 load-balancing manually as opposed to using an NGINX ingress. Assuming your services are deployed NodePort, you will need to: - create a load balancer instance - create a firewall rule that allows traffic from 130.211.0.0/22 and 35.191.0.0/16 to the K8 instance group - create a backend service against the K8 instance group, pointing to whatever node port the service in question exposes (they seem to be in the tcp:30000-40000 range, roughly; exact port is available from kubectl get service This is mildly error-prone in the sense that a service's exposed node port is presumably ephemeral wrt a kubectl replace, but still is much preferred to having to run a group of HAProxy/Nginx instances just to get around a temporary limitation in the NGINX ingress controller. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkQy3fsaoCwxC_EsYPglw3b4g9Vxocdks5tNUH8gaJpZM4P13dR> .

[lostpebble]

@zvozin I think you can set the NodePort to something static in the service descriptor - which should make this setup a little more solid.

I'm pretty much doing the same on my side, seems to be working quite well. And even though I'm not setting the NodePort statically as I should yet, it seems to be remembering them on full cluster resets.

Here are some notes I made during the process last time:

Linking NodePort service to the GCE Load Balancer

• Create the Service with NodePort
• Check the port number in kubernetes UI
• Create a health check for that port number e.g. HTTP 31331
  • Also set the url that the health check should hit like /_ah/health
  • Set timeout to something fair, like 4 seconds and the interval to something depending on how critical the service is
• After all that go the the Load Balancers area in Network Services
  • Click on “Advanced menu”, it’s small at the bottom (not the “Backends” menu at the top)
  • In advanced, go to “Back-end Services”
  • Create a new Backend Service
    • Set the named port to “port#####” e.g. “port31331”
    • Set the backend to the correct instance group
    • Set the port number there to the NodePort -> e.g. 31331
    • Max RPS is 1 RPS “for group”
    • Choose the health check that you just created
• Create the backend
• Open a firewall rule for the backend (https://stackoverflow.com/questions/42040238/how-to-expose-nodeport-to-internet-on-gce) - make sure the rule is on the network you're using in GCE if it's not the default
• Now you can select it in the regular Load Balancer edit interface and link to it as you please

[zvozin]

Thank you @scottefein and @lostpebble - good point!
Relevant docs: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

[adamsau]

+1

[lakgan]

+1. Just came across this issue. Would love to see buckets integrated

[bowei]

@mark-church @spencerhance

[yuranos]

Hey, how is it going with this one? Seems to be a very common usecase.

[anderspetersson]

Looks like the Ingress resource adds the underlying support for this in k8s 1.19:
https://github.com/kubernetes/kubernetes/pull/88775/commits
https://github.com/kubernetes/website/pull/21590/files#diff-56bb969f0c219fb744bcbcb43e40335a

[rick-pri]

I know that there's a kind of example here but does anyone have an actual working example for use on the GCP regular release channel?
The thing that I don't understand and I could find no documentation on is what the apiGroup is supposed to be.

[tarrencev]

Anyone having success configuring this?

[swetharepakula]

Even though Kubernetes v1 Ingress adds the ability to specify backends other than services the GCE Ingress controller does not support this.

[haizaar]

@swetharepakula do you know if it's planned and what's the timeline?

[haizaar]

Found this as well: https://issuetracker.google.com/issues/168854247

[rick-pri]

Even though Kubernetes v1 Ingress adds the ability to specify backends other than services the GCE Ingress controller does not support this.

Thanks for the response. I've since learned about the gateways API too which actually seems to be a much better fit for what we want to do. It looks like it's going to be a long time until we can use it.

[anderspetersson]

Any news on this now that 1.22 is available in GKE?

[bowei]

Hi all, while the backend references has been generalized, we need a K8s resource representation for buckets. There has been some work in the storage sig around a portable backend bucket representation which is one option, although that will take time to converge.

The other choice would be a CRD that is specific to GCP representing a reference to a bucket. It is a somewhat strange object as it is purely a reference, so we have to think carefully about the semantics around the object.

[dinvlad]

Sorry for spam, just wanted to mention for completeness that Service of type ExternalName also doesn't work in this case. Otherwise, it could provide a simple workaround by pointing the service to your bucket's domain name (after configuring it for static hosting).

[leedsjb]

+1 for this

[k725]

+1

[MarioIshac]

+1, would be very useful to serve both static (bucket) content and dynamic (service content) under one load balancer without it being overwritten by ingress.

[adamcathersides]

+1

[dides82]

+1, would be very useful to serve both static (bucket) content and dynamic (service content) under one load balancer without it being overwritten by ingress.

+1 would be great to see this kind of feature.
Wouldn't it be possible to leverage GKE config-connector capability in backend references of ingresses ? To have something like :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-backend
spec:
  defaultBackend:
    resource:
      apiGroup: storage.cnrm.cloud.google.com
      kind: StorageBucket
      name: my-bucket
  rules:
    - http:
        paths:
          - path: /icons
            pathType: ImplementationSpecific
            backend:
              resource:
                apiGroup: storage.cnrm.cloud.google.com
                kind: StorageBucket
                name: my-bucket

[swetharepakula]

With the introduction of the Gateway API, we will look to add this functionality with Gateway.

[gcbirzan]

I'm not entirely sure this 5+ year ticket being closed with "maybe we'll do it somewhere" warrants a completed status.

[swetharepakula]

Sorry, for the confusion. This was meant to be closed as not planned.

[MarioIshac]

Is there a ticket tracking the addition of this in the Gateway API? If not, @swetharepakula are you open to creating one w/ a timeline of when it can be added in the Gateway API? I understand why this ticket was closed, but the functionality is still very useful to have.

[omid-verneek]

It seems the Ingress only accepts Service and other custom resources such as ComputeBackendBucket are not accepted.