Track kubernetes resources by namespace and name

[manugarg]

While investigating #436, I found that we cache kubernetes resources by name, while they can have the same name across namespaces. We should use both -- namespace and name -- for caching kubernetes resources.

[jwillker]

@manugarg
There's another problem here. If the service is in a different namespace from the default, the Host header is sent incomplete.

I don't know, maybe something here?

cloudprober/probes/http/request.go

Line 54 in dd0bb3e

urlHost, hostHeader := target.Name, target.Name

Is sent only the service name(i think).

If you are using Istio this not work, you need to pass the entire FQDN like: service.namespace.svc.cluster.local

Because this cause upstream connect error

Look:

Without FQDN:

root@debug:/# curl  http://10.4.15.152:8800/healthz -v -H "Host:payout-http"
*   Trying 10.4.15.152...
* TCP_NODELAY set
* Connected to 10.4.15.152 (10.4.15.152) port 8800 (#0)
> GET /healthz HTTP/1.1
> Host:payout-http
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Wed, 05 Aug 2020 15:57:19 GMT
< server: envoy
<
* Connection #0 to host 10.4.15.152 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination

With FQDN:

root@debug:/# curl  http://10.4.15.152:8800/healthz -v -H "Host:payout-http.example-prod.svc.cluster.local"
*   Trying 10.4.15.152...
* TCP_NODELAY set
* Connected to 10.4.15.152 (10.4.15.152) port 8800 (#0)
> GET /healthz HTTP/1.1
> Host:payout-http.example-prod.svc.cluster.local
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< date: Wed, 05 Aug 2020 15:57:58 GMT
< x-envoy-upstream-service-time: 6
< server: envoy
<
* Connection #0 to host 10.4.15.152 left intact

Are causing problems in all targets.

I0805 14:35:45.535230       1 http.go:258] Target:payout-http:8800, URL:http://10.4.15.152:8800/healthz, response: upstream connect error or disconnect/reset before headers. reset reason: connection termination
I0805 14:35:45.535516       1 http.go:258] Target:risk-evaluator:5000, URL:http://10.4.0.38:5000/healthz, response: upstream connect error or disconnect/reset before headers. reset reason: connection termination
I0805 14:35:45.535792       1 http.go:258] Target:validation-app:80, URL:http://10.4.15.88:80/healthz, response: upstream connect error or disconnect/reset before headers. reset reason: connection termination
I0805 14:35:45.535969       1 http.go:258] Target:payout-http:8800, URL:http://10.4.15.152:8800/healthz, response: upstream connect error or disconnect/reset before headers. reset reason: connection termination

[manugarg]

@manugarg
There's another problem here. If the service is in a different namespace from the default, the Host header is sent incomplete.

I think that's an independent issue, but one that needs to be resolved. I've filed #438. To summarize briefly, it seems like need a way to specify host headers based on targets, for example, we should be able to do something like the following in probe config:

header {
  name: "Host",
  value:  "@target.name@. @[email protected]"
}

[manugarg]

@JohnWillker Duplicate resources causing HTTP probe issue should be resolved for you now. I am closing this. Feel free to re-open if the issue is still not fixed for you.

I'll follow up on #438 independently.

[jwillker]

@JohnWillker Duplicate resources causing HTTP probe issue should be resolved for you now. I am closing this. Feel free to re-open if the issue is still not fixed for you.

I'll follow up on #438 independently.

@manugarg I have tested, and the metrics are working perfectly now. Thanks again for the fast resolution.