Add health checking from NGINX to individual endpoints. #146
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
This change is |
chrismoos
force-pushed
the
nginx_health_check
branch
from
7d1eac3
to
9867801
Compare
|
@bprashanth I've added in the changes here to nginx that were made on kubernetes-retired/contrib#2329. |
|
Is there any sort of integration or unit tests that could be added here? Not sure if it is plugin specific. |
| @@ -365,6 +372,12 @@ http { | |||
| access_log off; | |||
| return 200; | |||
| } | |||
|
|
|||
| {{ if $cfg.UseUpstreamHealthChecks }} | |||
| location /upstream_status { | |||
There was a problem hiding this comment.
Please add an example, as in this pr: #235, there's already an examples/health-checks directory, just create a gce/ and nginx/ under that and add it under nginx/.
Please also mention in that example what this endpoint exposes.
| @@ -302,6 +308,19 @@ func newIngressController(config *Configuration) *GenericController { | |||
| glog.Warning("Update of ingress status is disabled (flag --update-status=false was specified)") | |||
| } | |||
|
|
|||
| ic.podLister.Indexer, ic.podController = cache.NewIndexerInformer( | |||
There was a problem hiding this comment.
Fyi this will increase the memory footprint of the controller in large clusters. Somethig to keep in mind, not much we can do about it at this point I guess. We can think about optimizing that when the time comes, however we might have to update examples that have request/limits set (unsure if there are any).
| if err != nil { | ||
| return nil, fmt.Errorf("Failed to get pod listing: %v", err) | ||
| } | ||
| for _, pod := range pods { |
There was a problem hiding this comment.
I guess you want this to deterministically give you the same health check, even if there are 3 endpoint pods behind the service with all different readiness probes, we should sort based on creation timestamp and always take the first one.
| if servicePort.Type == intstr.Int && int(port.ContainerPort) == servicePort.IntValue() || | ||
| servicePort.Type == intstr.String && port.Name == servicePort.String() { | ||
| if container.ReadinessProbe != nil { | ||
| if container.ReadinessProbe.HTTPGet == nil || container.ReadinessProbe.HTTPGet.Scheme != "HTTP" { |
There was a problem hiding this comment.
Any reason to not support https health checks? (I'm fine with the HTTP caveat, just curious)
| @@ -102,6 +102,16 @@ type BackendInfo struct { | |||
| Repository string `json:"repository"` | |||
| } | |||
|
|
|||
| // UpstreamCheck is used to configure ingress health checks | |||
| type UpstreamCheck struct { | |||
| HttpSend string | |||
There was a problem hiding this comment.
please add a godoc comment for each public field, especially HttpSend, Fall and Rise which are non-obvious.
Also please s/Http/HTTP in conformance with https://github.com/golang/go/wiki/CodeReviewComments#initialisms
|
Also please add unittests |
|
Ideally a unittest might seed the pod cache with some fake pods and check to see that the right health check is retrieved with int/named ports. Let me know if you need me to dig up example unittests where we do something similar. @cmluciano integration tests are a little harder, we're working on something generally applicable cross kube-repo, but till then we mostly rely on e2es and unittests. The ingress e2es already point at an rc with a readiness probe, so we can probably modify that to fail if this feature doesn't work once the image is published: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/ingress/http/rc.yaml#L18 |
|
|
Is this still under active development? Would be very useful for us, I'd be happy to lend a hand if it's helpful. |
|
See #981, this is probably preferable to this as unfortunately this duplicates the |
Overview
This change adds support for the nginx_upstream_check_module and configures upstreams with health checks based on the readinessProbe in the service's pod(s).
It is configurable with ConfigMap (default disabled):
use-upstream-health-checksIf the upstream health checking is enabled there is also an additional endpoint at /upstream_status.
Notes
This was originally proposed in the old NGINX ingress controller @ kubernetes-retired/contrib#2072
This change depends on a new
nginx-slimimage that would target version 0.13. The PR for that is kubernetes-retired/contrib#2329A request for this was also raised in #140.