More and better documentation needed for k3os.k3s_args

[pagong]

Is your feature request related to a problem? Please describe.
I've spent several days, trying to get k3s options, that had worked with k3os <= v0.2.1, working again with k3os >= v0.3.0. I was struggling with cluster-cidr and service-cidr options, that take an additional value as argument. But this relates to all k3s options that take additional values.

Describe the solution you'd like
The following config.yaml does no longer work:

hostname: rancher225
k3os:
   k3s_args:
  - server
  - "--cluster-cidr 10.107.0.0/23"
  - "--service-cidr 10.107.1.0/23"

k3s does not start up properly and logs strange error messages to /var/log/k3s-service.log.

Describe alternatives you've considered
The following config.yaml using '=' does work, and should be documented:

hostname: k3os060a
k3os:
  environment:
    http_proxy: http://192.168.107.9:8080
    https_proxy: http://192.168.107.9:8080
    no_proxy: localhost,127.0.0.1,192.168.107.0/24
  k3s_args:
  - server
  - --cluster-cidr=10.107.6.0/23
  - --service-cidr=10.107.7.0/23

The following config.yaml using separate lines and double quotes does work:

hostname: k3os071a
k3os:
  environment:
    http_proxy: http://192.168.107.9:8080
    https_proxy: http://192.168.107.9:8080
    no_proxy: localhost,127.0.0.1,192.168.107.0/24
  k3s_args:
  - server
  - "--cluster-cidr"
  - "10.107.4.0/23"
  - "--service-cidr"
  - "10.107.5.0/23"

And the following config.yaml using separate lines and no quotes does work, and should be documented:

hostname: rancher233
k3os:
  environment:
    http_proxy: http://192.168.107.9:8080
    https_proxy: http://192.168.107.9:8080
    no_proxy: localhost,127.0.0.1,192.168.107.0/24
  k3s_args:
  - server
  - --cluster-cidr
  - 10.107.2.0/23
  - --service-cidr
  - 10.107.3.0/23

Additional context
Thanks to the Rancher folks for clarifying the k3s_args options to me.

[dweomer]

The following config.yaml does no longer work:

hostname: rancher225
k3os:
   k3s_args:
  - server
  - "--cluster-cidr 10.107.0.0/23"
  - "--service-cidr 10.107.1.0/23"

If the --cluster-cidr and --service-cidr (or any other CLI) flags ever worked as specified above then that was a bug. This is because k3s_args is an exec-style (aka uninterpreted) argument array, this means that k3os is invoking k3s like so:

#!/bin/sh
exec "k3s" "server" "--cluster-cidr 10.107.0.0/23" "--service-cidr 10.107.1.0/23"

When what was really meant was (notice the space between the cidr flags and their values):

#!/bin/sh
exec "k3s" "server" "--cluster-cidr" "10.107.0.0/23" "--service-cidr" "10.107.1.0/23"

When we run the below in bash:

"k3s" "server" "--cluster-cidr 10.107.0.0/23" "--service-cidr 10.107.1.0/23"  | tail -1

we get:

FATA[0000] flag provided but not defined: -cluster-cidr 10.107.0.0/23

Which is k3s complaining that a flag was specified that it doesn't understand because we have passed --cluster-cidr 10.107.0.0/23 as a single argument and not two. There is a UNIX-y Go-ism that allows us to pass flag values as single arguments, that being the --flag-name=value syntax, and this alternative syntax could be better documented with examples.

---

See also:

• https://golang.org/pkg/flag/#hdr-Command_line_flag_syntax
• https://docs.docker.com/engine/reference/builder/#cmd
• https://docs.docker.com/engine/reference/builder/#entrypoint
• https://linux.die.net/man/3/exec

[dweomer]

Addressed by #373, thanks @jdbohrman

[pagong]

Thank you, @jdbohrman and @dweomer.