Development proxy ignores path in K8s API URL

[rbq]

Bug Report

What did you do?

1. set ~/.kube/config to a Rancher-managed cluster's API
   (sth. like https://my-domain.tld/k8s/clusters/c-some-id)
2. kubectl apply […] to deploy CRD
3. operator-sdk run to run operator

What did you expect to see?

I expected both steps 2 and 3 to target the cluster configured in step 1.

What did you see instead? Under which circumstances?

1. kubectl apply works as expected
2. operator-sdk run ignores the path component when setting up the proxy and uses https://my-domain.tld instead

In my particular case this didn't actually surface as an error. Instead the modified API endpoint without a path happened to target the management cluster of my Rancher deployment instead of a development cluster. So it actually ran the operator against a different cluster without any warning at all.

Environment

Operator type:

/language ansible

Kubernetes cluster type:

Rancher, RKE

$ operator-sdk version

operator-sdk version: "v1.7.1-19-g3ce1eeef", […], kubernetes version: "v1.19.4", […]

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"21", […]}
Server Version: version.Info{Major:"1", Minor:"20", […]}

Possible Solution

1. To prevent people from shooting themselves in the foot I'd suggest that operator-sdk run (and any other commands that might behave similarly) fail with an error message if they encounter an API endpoint with a path component.
2. It would be nice if Operator SDK could be improved to work with clusters that have an API with a path component, such as the ones behind a Rancher instance or other proxies/multiplexers.

Additional context

• kubectl proxy not working for kube contexts with base path kubernetes/kubernetes#94280
• https://kubernetes.slack.com/archives/CAW0GV7A5/p1620896783095000

[laxmikantbpandhare]

Fix for this issue is dependent on kubectl PR. Once this PR is merged then only we will be able to move ahead with this fix as it has changes in apimachinery.

[laxmikantbpandhare]

Created another issue #4980 which will fix this issue and throw an error for now. Once kubectl PR gets merged then we will move ahead with changes.

[rashmigottipati]

Removing this from v1.9.0 milestone as #4980 (in milestone v1.9.0) is tracking this issue.

[laxmikantbpandhare]

As part of the Operator SDK v1.9.0 release, we have moved temporary fix for this issue. It will throw an error if API endpoint has Path in it. @rbq Could you please try with new version of Operator SDK. You can download the release and view the changes here: https://github.com/operator-framework/operator-sdk/releases/tag/v1.9.0

[rbq]

As part of the Operator SDK v1.9.0 release, we have moved temporary fix for this issue. It will throw an error if API endpoint has Path in it. @rbq Could you please try with new version of Operator SDK. You can download the release and view the changes here: https://github.com/operator-framework/operator-sdk/releases/tag/v1.9.0

Checked, throws an error as expected. :)

[laxmikantbpandhare]

@rbq - thank you for your confirmation :)

[laxmikantbpandhare]

Till now, we did the temporary fix for this issue as this issue was dependent on kubectl PR. kubectl PR got merged and we are moving ahead with the permanent fix for this issue.

[laxmikantbpandhare]

@rbq - I wanted to re-create this issue on my local machine. Wanted to know about the steps you did. I am working RKE cluster for the first time.

[rbq]

@laxmikantbpandhare I don't know if you can actually use your local machine to run both Rancher and a cluster node, but the steps should be:

1. run Rancher as a single container
2. provision a cluster via Rancher
3. download the kubeconfig file

The kubeconfig file will instruct clients to access the provisioned cluster through Rancher using a URL that contains a path.

[laxmikantbpandhare]

@laxmikantbpandhare I don't know if you can actually use your local machine to run both Rancher and a cluster node, but the steps should be:

1. run Rancher as a single container
2. provision a cluster via Rancher
3. download the kubeconfig file

The kubeconfig file will instruct clients to access the provisioned cluster through Rancher using a URL that contains a path.

Great, I will try this. thank you.

[laxmikantbpandhare]

The upstream PR got merged in Mid August and Kubectl had 1.22 release. But, in 1.22 release, Kubectl did not released the change required for this fix. I can see it is now added in 1.23-alpha release and it is likely to get release in 1.23. Changelog for the upcoming release of Kubectl - https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md - PR - 97350

[jmrodri]

This was waiting on k8s 1.23 release per the above comment. We merged k8s 1.23 in PR #5505

[rbq]

I'm still getting the message "[...] contains a path component, which the proxy server is currently unable to handle properly. Work on this issue is being tracked here: [...]" in 1.19.0.

[jmrodri]

/cc @jmrodri

[laxmikantbpandhare]

thank you for bringing this @rbq - We are working on it and will fix this soon.

[laxmikantbpandhare]

Raised above PR that will remove the content and will not throw an error if the proxy server contains a path in it.

[laxmikantbpandhare]

@rbq - We removed the change that was throwing an error when there is a path in the proxy server. It will be available in the 1.20 operator-sdk release.

[rbq]

@laxmikantbpandhare That's great to hear, thanks a lot!

[rbq]

@laxmikantbpandhare Are releases cut based on schedule (the milestone says due today) or features?

[laxmikantbpandhare]

@rbq - We planned to do it by EOD tomorrow. I will update you here.

[laxmikantbpandhare]

@rbq - We have released Operator SDK v1.20.0. Could you please verify and let us know your observations.

[rbq]

@laxmikantbpandhare Yay, it's been merged into Homebrew formulas and does seem to work!

[laxmikantbpandhare]

@rbq - that's great, thank you for the confirmation.