kubectl v1.29.3 ignores HTTPS_PROXY environment variable

[kriswuollett]

What happened:

I am unable to connect to the API via a socks5 proxy via the HTTPS_PROXY environment variable. Instead it seems like only works when I add the same value to the proxy-url property in the cluster config.

What you expected to happen:

I expected to be able to use kubectl via a HTTPS proxy environment variable instead of the proxy-url attribute according to the documentation:

To access the Kubernetes API server through the proxy you must instruct kubectl to send queries through the SOCKS proxy we created earlier. Do this by either setting the appropriate environment variable, or via the proxy-url attribute in the kubeconfig file.

How to reproduce it (as minimally and precisely as possible):

• get a working kubernetes cluster with API server on a remote host (I used kind)
• run a command like ssh -D 8002 -q -N remote-host to set up a socks proxy from localhost to remote-host
• try running a command like HTTPS_PROXY=socks5://localhost:8002; export HTTPS_PROXY; kubectl --context kind-remote-host get nodes

Anything else we need to know?:

The ability to use HTTPS_PROXY environment variable is extermely useful since it avoids hard-coding a port number to use for proxy access. The proxy-url attribute assumes one would want to statically assign different port numbers across numerous clusters your user account may wish to access.

Environment:

Client:

Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
macOS 14.6.1 (23G93)

Remote cluster:

kind version 0.24.0
debian 12 x86_64

[ardaguclu]

Interesting, could you please try;

HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-remote-host

also run this command with high verbosity to see;

HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-remote-host -v=9

[ardaguclu]

I can see that it is set by https://github.com/kubernetes/kubernetes/blob/3f3e1d507d081528ce8081f0daf4446d8a3b9bc0/staging/src/k8s.io/client-go/transport/cache.go#L125

[kriswuollett]

Why do I see curl in the log, shouldn't there be a native go HTTP client?

Without proxy-url attribute

% HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-nyc3-shared -v=9; kubectl version                   
I0912 19:59:50.425443   98668 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 19:59:50.430781   98668 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 19:59:50.432117   98668 round_trippers.go:508] HTTP Trace: Dial to tcp:127.0.0.1:34487 failed: dial tcp 127.0.0.1:34487: connect: connection refused
I0912 19:59:50.432133   98668 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500  in 1 milliseconds
I0912 19:59:50.432137   98668 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 1 ms
I0912 19:59:50.432140   98668 round_trippers.go:577] Response Headers:
I0912 19:59:50.432344   98668 helpers.go:264] Connection error: Get https://127.0.0.1:34487/api/v1/nodes?limit=500: dial tcp 127.0.0.1:34487: connect: connection refused
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?

Added proxy-url back to the context

It doesn't explictly say a proxy is being used other than the dial and curl port numbers are different.

% kubectl get nodes --context kind-nyc3-shared -v=9           
I0912 20:00:45.645722   98714 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 20:00:45.657230   98714 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 20:00:45.662387   98714 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost resolved to [{::1 } {127.0.0.1 }]
I0912 20:00:45.663187   98714 round_trippers.go:510] HTTP Trace: Dial to tcp:[::1]:8002 succeed
I0912 20:00:46.338098   98714 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500 200 OK in 680 milliseconds
I0912 20:00:46.338163   98714 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 233 ms ServerProcessing 223 ms Duration 680 ms
I0912 20:00:46.338175   98714 round_trippers.go:577] Response Headers:
I0912 20:00:46.338188   98714 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 0b411ff5-0cbc-4da2-b4c5-b8ca36074eb1
I0912 20:00:46.338197   98714 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 00fd2612-1189-46b0-b59b-2aa96bfc524b
I0912 20:00:46.338205   98714 round_trippers.go:580]     Date: Thu, 12 Sep 2024 12:00:45 GMT
I0912 20:00:46.338212   98714 round_trippers.go:580]     Audit-Id: 3f7f582a-753e-4b65-8059-88535c9b5cbe
I0912 20:00:46.338220   98714 round_trippers.go:580]     Cache-Control: no-cache, private
I0912 20:00:46.338228   98714 round_trippers.go:580]     Content-Type: application/json

...

[ardaguclu]

Sounds like an issue and requires investigation.

run a command like ssh -D 8002 -q -N remote-host to set up a socks proxy from localhost to remote-host

What is the exact command configuring this on Kind cluster?

[ardaguclu]

/triage accepted
/priority backlog
/assign

[kriswuollett]

There is no proxy information on the remote Kind cluster. Basically:

• create Kind cluster on remote host
• copy the kube config from the remote host to local host
• run ssh -D 8002 -q -N remote-host in a separate terminal on local host (where remote-host is the name of a ssh config that has private key, IP address, etc.
• try kubectl get nodes with the HTTPS_PROXY environment variable, it fails
• insert the proxy-url into the kube config either by hand, or when I have some automation with scripts, I use the jq query '.clusters.[0].cluster |= ({"proxy-url": "socks5://localhost:8002"} + .)'
• try kubectl get nodes again, it works without HTTPS_PROXY

Don't think it is relevant, but here is my Kind config file:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: nyc3-shared
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

And I created it with the command kind create cluster --config /usr/local/etc/kind.yaml --wait 5m. Its running on a DigitalOcean droplet with debian-12-x64 image.

[kriswuollett]

I also tried on a Ubuntu 22.04 x86_64 host of mine:

$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.0

It has the same behavior as above on my mac. Also tried a more recent kubectl build with no success:

$ ./kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.0

[ardaguclu]

Thank you. I don't have remote host to test this and I need to find a way to reproduce this just on my local.

[xyz-li]

Why do I see curl in the log, shouldn't there be a native go HTTP client?

Without proxy-url attribute

% HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-nyc3-shared -v=9; kubectl version                   
I0912 19:59:50.425443   98668 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 19:59:50.430781   98668 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 19:59:50.432117   98668 round_trippers.go:508] HTTP Trace: Dial to tcp:127.0.0.1:34487 failed: dial tcp 127.0.0.1:34487: connect: connection refused
I0912 19:59:50.432133   98668 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500  in 1 milliseconds
I0912 19:59:50.432137   98668 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 1 ms
I0912 19:59:50.432140   98668 round_trippers.go:577] Response Headers:
I0912 19:59:50.432344   98668 helpers.go:264] Connection error: Get https://127.0.0.1:34487/api/v1/nodes?limit=500: dial tcp 127.0.0.1:34487: connect: connection refused
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?

If the apiserver address is https://127.0.0.1:34487, kubectl (specially go runtime) will ignore HTTPS_PROXY.
According to the implementation of golang. It ignores HTTP_PROXY/HTTPS_PROXY when dest IP is localhost or loopbak.
https://github.com/golang/go/blob/master/src/vendor/golang.org/x/net/http/httpproxy/proxy.go#L169-L202
I don't think this is a bug.

[ardaguclu]

Thanks @xyz-li. In that case how does proxy-url work?

[kriswuollett]

If kubectl has a command line option for things like insecure-skip-tls-verify, then why not add proxy-url as a kubectl command line parameter, which would override any setting in the config, to solve this issue? That way the whole issue of the environment variable usage not really being a standard can be dropped.

[xyz-li]

Thanks @xyz-li. In that case how does proxy-url work?

https://github.com/kubernetes/kubernetes/blob/359b9ba9bf5c508aeee0eee520d09b7bf1fdaa50/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L198-L206
@ardaguclu

[ardaguclu]

As there is no bug, I think we can close this issue. @kriswuollett the ask you above seems like a feature request and can be handled in a separate issue.

/close

[k8s-ci-robot]

@ardaguclu: Closing this issue.

In response to this:

As there is no bug, I think we can close this issue. @kriswuollett the ask you above seems like a feature request and can be handled in a separate issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.