k0s kubeconfig ignores config value for the external address

[mikhail-sakhnov]

Version

$ k0s version

Current main head

Platform
Which platform did you run k0s on?

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

What happened?
I use following cluster configuration with set up external address:

k0s kc -n kube-system get ClusterConfig -oyaml
apiVersion: v1
items:
- apiVersion: k0s.k0sproject.io/v1beta1
  kind: ClusterConfig
  metadata:
    creationTimestamp: "2021-10-20T10:52:18Z"
    generation: 1
    name: k0s
    namespace: kube-system
    resourceVersion: "223"
    uid: 185d9534-264f-4084-b29d-3e7d9d2c63cf
  spec:
    api:
      address: 10.0.0.24
      externalAddress: ec2-13-49-159-141.eu-north-1.compute.amazonaws.com
      port: 6443
    network:
      dualStack: {}
      kubeProxy:
        mode: iptables
      kuberouter:
        autoMTU: true
        mtu: 0
        peerRouterASNs: ""
        peerRouterIPs: ""
      podCIDR: 10.244.0.0/16
      provider: kuberouter
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Whenever I use k0s kubeconfig command to have kubeclient config, either for admin or by using k0s kubeconfig create username, I expect to see external address in the returned kubeconfig, but I get the internal address (10.0.0.24).

k0s kubeconfig create user

apiVersion: v1
clusters:
- cluster:
    server: https://10.0.0.24:6443
    certificate-authority-data: ...
  name: k0s
contexts:
- context:
    cluster: k0s
    user: user
  name: k0s
current-context: k0s
kind: Config
preferences: {}
users:
- name: user
  user:
    client-certificate-data: ...
    client-key-data: ...```

**How To Reproduce**
Use the steps from the description

**Expected behavior**
I expect kubeconfig to have external address as a value for the `cluster.server`

**Additional context**
Add any other context about the problem here.

[trawler]

This issue stems from an old problem that the dynamic configuration feature was supposed to fix, but with the current implementation still needs to be addressed.

Background

k0s configuration can currently be fed into k0s in two ways:

• by feeding in a configuration yaml file with the --config-file command line option
• by leveraging the dynamic configuration customResource.

Since dynamic configuration is disabled by default, the configuration is then fed to k0s using the 1st method, and since this configuration is not persistent, the config-file parameter needs to be added to each and every command.

Default Values

If a configuration file is not provided, k0s will generate a default configuration, identical to what is provided when running the k0s default-config command line.

Root Cause

This problem above is not directly related to dynamic configuration at all. It is caused by starting the cluster with a custom externalAddress configuration (I.E. custom config file) and not running the k0s kubeconfig command with --config-file option. In which case, the k0s kubeconfig command is then generating a default configuration. k0s actually behaves as designed, but this design is broken.

Config Parsing

Currently, for almost every k0s command we run, the following logic applies:

1. Search for the config-file flag and read the yaml file contents.
2. If none exists, generate a default config struct, and use that.
3. If dynamic configuration is enabled, get the configuration from the API and merge it with the configuration generated by one of the previous two steps.

This logic is flawed, since it still requires the user to feed in the custom configuration file for each k0s command line option. If the user forgets, the outcome is unpredictable.

I would like to suggest that we:

• Stop generating default values for each and every command.
• Set up a single source of configuration truth, whether it is a yaml file, a default config object or a ClusterConfig CR.
• In the case of a default config file, or a custom config file, store the file persistently.

Mitigation Options

The requirement to feed in a config-file for each command is problematic, and is prone to user mistakes.
I think a better solution would be to read-in whatever configuration option is fed during k0s controller or k0s worker initial run, and persist it, until the next time controller or worker are started.
Following a discussion in the team, two main mitigation options were suggested. I will list them below, and compare them with three main criteria: persistency, transparency, and user experience.

Runtime Config file

1. Persistency: During startup, when a custom config-file is fed to the k0s controller or k0s worker command, k0s will read the file, and write a "persistent" config-file under a runtime directory (E.g./run/k0s). During subsequent runtime commands, k0s will no longer refer to the original configuration file, but will only refer to the runtime file.
2. Transparency: This options is transparent to the user, and there is no need to update the documentation.
3. User experience: Users who are already familiar with k0s will find this option relatively intuitive, as it will not require them to update the existing startup scripts. Configuration file location is flexible and does not require special root access privileges.

System-wide Config file

1. Persistency: With this option, k0s can be started without using the config-file option at all. K0s system configuration file will be defined as /etc/k0s/k0s.yaml.
2. Transparency: Documentation will have to be updated to reflect the new change. This change will probably also have impact on k0s install and k0s reset commands.
3. User experience: Users will need to modify their startup scripts. Probably remove the config-file flag, but also make sure they have access to the central configuration location (I.E. root level access).
4. OS standards: by defining a central location for the configuration file, k0s will better conform with linux binary standards (but we may also need to verify that all linux flavours use the above mentioned location as defaults).

[ncopa]

The idea with system-wide config is to change the docs to say something like:

Generate a yaml config file that uses the default settings.

k0s default-config > /etc/k0s/k0s.yaml

...
Install k0s with your new config file.

sudo k0s install controller -c /etc/k0s/k0s.yaml

Which basically means that the docs provides an opinion to where the config file should be instead of saying /path/to/your/config/file.

Then we change the logic in k0s to use config from:

1. from config API (custom resource).
2. if config does not exist as a custom resource it will fall back to --config-file
3. if --config-file is not specified, it will look for and use ./k0s.yaml.
4. if ./k0s.yaml does not exist it will look for and use the default system config file, /etc/k0s/k0s.yaml
5. if /etc/k0s/k0s.yaml does not exist, it will fall back to defaults.

This will not solve the problem for current installs which have the config file in custom location (eg /path/to/your/config/file) and not as /etc/k0s/k0s.yaml. To solve the issue for those users, they will have to manually move the file to /etc/k0s/k0s.yaml and edit the systemd unit file or openrc script.

However it will solve the issue for new and future installs, using the doc.

We could optionally also modify k0s install to copy the provided k0s.yaml to /etc/k0s/k0s.yaml (unless the provided config is that spcific path) and make the startup script use --config-file /etc/k0s/k0s.yaml. I don't think k0s reset needs to delete it (or we could have an option --keep-config).

[ncopa]

Runtime Config file

1. Persistency: During startup, when a custom config-file is fed to the k0s controller or k0s worker command, k0s will read the file, and write a "persistent" config-file under a runtime directory (E.g./run/k0s). During subsequent runtime commands, k0s will no longer refer to the original configuration file, but will only refer to the runtime file.
2. Transparency: This options is transparent to the user, and there is no need to update the documentation.
3. User experience: Users who are already familiar with k0s will find this option relatively intuitive, as it will not require them to update the existing startup scripts. Configuration file location is flexible and does not require special root access privileges.

As an option to this we could use /var/lib/k0s/ instead of /run/k0s, and the config file would survive reboots if we want that. It would mean that config would be generated and stored at first startup only, instead of each time k0s starts up. I don't know if we need to be able fish out config options (like externalAddress) while the controller is in a stopped state.

[trawler]

config would be generated and stored at first startup only, instead of each time k0s starts up.

Wouldn't that mean that a user would only be able to update the configuration by editing /var/lib/k0s/k0s.yaml?
I suspect that would be more confusing to users (they would have to dig-up the documentation to figure out where the file is stored).

[ncopa]

config would be generated and stored at first startup only, instead of each time k0s starts up.

Wouldn't that mean that a user would only be able to update the configuration by editing /var/lib/k0s/k0s.yaml? I suspect that would be more confusing to users (they would have to dig-up the documentation to figure out where the file is stored).

correct. yes, this may be confusing. I'm just throwing out ideas here, including bad ideas 😄

We could overwrite this file every time k0s starts, similar to what we'd need to do with /run/k0s. The difference is that /var/lib/k0s survives reboots while /run does not. Thus the question: do we need to be able to run any cli commands (eg k0s kubeconfig) after a reboot and before k0s starts up? If yes, then we cannot use /run.

[trawler]

On a discussion with @jnummelin, we've come up with sort of a merge of the two approaches listed above:

1. If a custom config-file is used during controller startup, a symlink to this config-file will be created under /run/k0s. Subsequent command runs will refer to the file listed in the symlink. If the symlink file cannot be read, an error is printed and the command fails.
2. When no config-file parameter is fed during the controller startup:
   • k0s will look for a default config-file under /etc/k0s/k0s.yaml. If it can find one, it will create a symlink under /run/k0s (similar to the behaviour listed above).
   • If none, exists, default configuration file will be used.
     Subsequent k0s command runs, will either refer to the symlinked file, or use a default config if no symlink exists.

[jnummelin]

The issue is marked as stale since no activity has been recorded in 30 days

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days