Add spec.api.onlyBindToAddress configuration

[pschichtel]

I picked up #1038 and rebased it onto main. Smoke tests pass, but not entirely clear what was left here.

From #1038:

Should a new bindAddress option be created and change documentation for the existing address option, or should the existing address option's behavior be changed to match the existing documentation?

The later I can see causing some issues with existing setups, bindAddress is probably the safest route. Currently the default for bindAddress is "0.0.0.0", which is the default kube-api-server uses if the --bind-address flag isn't used.

Issue Fixes #957

What this PR Includes Adds the ability to define a bindAddress for the kube-api-server.

[twz123]

Thanks for picking this up.

I'd double-check if there are some bad interactions with node-local load balancing? I imagine the LB config needs to pick up the correct bind addresses as well?

We should think about some integration test, too, I suppose.

[twz123]

Okay, some things to proceed with:

• We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.
• We need to double-check if this plays nicely with nllb and other things. Do we need to update the docs on e.g. external load balancing? How would this affect k0sctl, if at all?

[pschichtel]

We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.

I'm not very familiar with the code base yet, can you point me to examples for integration tests?

We need to double-check if this plays nicely with nllb and other things.

I'd have to look into how nllb works exactly. Depending on how the upstream connection is made its configuration probably needs adjustments.

Do we need to update the docs on e.g. external load balancing?

I guess a disclaimer like "if you change the bind address make sure to align your external load balancer" or so. I feel like people changing bind addresses are probably aware of the impact. But that's just my opinion.

How would this affect k0sctl, if at all?

k0sctl would need to interact with the API server on the correct IP, wouldn't it?

[twz123]

Sorry for getting back to you so late @pschichtel. I'll bluntly blame the holiday season for that. 😅

We need an integration test. Need to come up with a scenario for that, i.e. what is to be tested.

I'm not very familiar with the code base yet, can you point me to examples for integration tests?

We need to double-check if this plays nicely with nllb and other things.

I'd have to look into how nllb works exactly. Depending on how the upstream connection is made its configuration probably needs adjustments.

We could try to start with the existing nllb inttest, copy that over into a separate directory and start hacking on the new one. You can probably delete all the cluster-restart stuff in there, and just rely on the cluster bootstrapping part. You need to add the new directory name to inttest/Makefile.variables. Then you should be able to do something like make check-yolo (or whatever the name is you choose.)

Might make sense to simply try to overwrite the bind addresses of every component that you can get hold of and see what happens to the cluster.

Do we need to update the docs on e.g. external load balancing?

I guess a disclaimer like "if you change the bind address make sure to align your external load balancer" or so. I feel like people changing bind addresses are probably aware of the impact. But that's just my opinion.

Sounds good enough for the start.

How would this affect k0sctl, if at all?

k0sctl would need to interact with the API server on the correct IP, wouldn't it?

Yes. Would it need to look at the k0s config to figure this out? Or would it "just work"? Maybe @kke can tell.

[pschichtel]

Sorry for getting back to you so late @pschichtel. I'll bluntly blame the holiday season for that. 😅

No worries, I don't have any rush with this one.

We could try to start with the existing nllb inttest, copy that over into a separate directory and start hacking on the new one. You can probably delete all the cluster-restart stuff in there, and just rely on the cluster bootstrapping part. You need to add the new directory name to inttest/Makefile.variables. Then you should be able to do something like make check-yolo (or whatever the name is you choose.)

Might make sense to simply try to overwrite the bind addresses of every component that you can get hold of and see what happens to the cluster.

Makes sense, I'll have a go at that.

[twz123]

(The linter error will go away after a rebase. See #3991 for details.)

[pschichtel]

yeah I plan to work on this again towards the end of this week

[andycandy-de]

Any idea when this feature will be available? I also have multiple IPs and only want to bind the k0scontroller to a specific IP.

[pschichtel]

@andycandy-de sadly this had to be pushed back a little in my prio list, but I still intent to eventuelly finish this up. I think some of the recent k0s/k8s releases also added additional components, which probably needs/deserves some investigation. I'm currently unable to work on it until mid of April, so no sooner than that.

[pschichtel]

Great. I'm currently on the go, but can probably clean this up on Sunday.

[github-actions]

This pull request has merge conflicts that need to be resolved.

[github-actions]

This pull request has merge conflicts that need to be resolved.

[pschichtel]

Is that failing job just flakyness? It seems to exceeded a deadline unrelated to my change, right?

[twz123]

Yep, the inttests are quite a bit flaky, unfortunately.

[twz123]

Thank you!

[mback2k]

k0sctl will need to be tweaked to support this as it currently tries to validate the API server coming up this way:

executing `curl -kso /dev/null --connect-timeout 20 -w \"%{http_code}\" \"https://localhost:6443/version\"`"

and then fails this way:

INFO ==> Running phase: Upgrade controllers 
INFO [ssh] ...: starting upgrade   
INFO [ssh] ...: waiting for the k0s service to start 
INFO * Running clean-up for phase: Acquire exclusive host lock 
INFO * Running clean-up for phase: Upgrade controllers 
INFO ==> Apply failed 

[twz123]

@mback2k Mind creating a k0sctl issue for that?