[Design Doc] feat: new configuration system + config subcommand

[phm07]

Glossary

Option

An option is a key-value pair that modifies how the CLI behaves.

Examples:

• Default SSH Keys
• Debug Logging Enabled
• Current Active Context
• API Token

Context

A context is a named list of options that the user can enable to apply all options defined in it. It always includes an API Token. Only a single context can be active at a time.

Configuration File

The configuration file contains a collection of contexts and global options. It is being read by the CLI.

Preferences

The subset of options that are not mandatory to configure before the CLI can be used.

Configuration

All the ways the user can set options for the CLI. Including the Configuration File, Environment Variables and Flags.

Motivation

Our CLI already uses a configuration file to save the available contexts (API Tokens + Name) as well the current selected context. Modifying the file is done through sub-commands hcloud context create|delete|use|active|list

• The CLI also has some global options exposed through environment variables
  • HCLOUD_DEBUG
  • HCLOUD_TOKEN
  • HCLOUD_CONFIG
  • HCLOUD_ENDPOINT
  • HCLOUD_DEBUG_FILE
  • HCLOUD_CONTEXT
• The CLI also has some global options exposed through flags
  • --poll-interval
  • --quiet

The config is saved to a file in (UNIX) ~/.config/hcloud/cli.toml.

The current options are badly/not documented. It is not clear where you can set which options.

• We are currently thinking about adding further config options
  • Default SSH keys
  • Table sort
  • Table columns
  • Default location
  • Update Checks

This Design Doc is intended to find a good solution that can handle the current and future options while keeping ways of configuration consistent.

Implementation

What are Options?

• Key/Value pairs
• Keys are always strings in kebab-case.
• Keys can be namespaced with .
• Values can have these data types for the :
  • string, integer, float, bool
  • Any of the above as array

UX / How does the user interact with the configuration file?

We will introduce a new hcloud config sub-command that provides a generic interface to modify options (from the preferences) in the configuration file.

Usages:

• hcloud config --help
  • Explains all options and their default values
• hcloud config [--global] get <key>
  • Shows the current value of key
  • In the future: show origin of value with --show-origin flag (config, env, flags…)
• hcloud config [--global] set <key> <value>...
  • Set/Replace the value of a key
• hcloud config [--global] unset <key>
  • Delete an option from the config
• hcloud config [--global] <add|remove> <key> <value>...
  • Only applies to array options. Adds or removes the values from the key.
• hcloud config [--global] list
  • Lists all configured options

We plan to add autocomplete for option keys. For option values we want to add autocomplete if possible. This can be done with Cobra’s ValidArgsFunction.

By default, this command modifies the options saved in the current active context. Changing this can be done through the context option (--context, HCLOUD_CONTEXT). To change options on the global scope, the user can specify the --global flag.

Using --global as opt-in, we prevent accidental options overwrites.

How to deal with existing flags?

If the option defines a default that already has one or multiple flags (like default-ssh-keys in hcloud server create [--ssh-key]) we do not add new flags for these options. The existing flag would default to the option if the option is set. If the actual flag was set, the default option is overridden.

Options that are not “defaults for existing flags” should get their own flag at the root command level.

For other global existing Environment Variables and Flags we will try to unify them into the new configuration pattern.

How do we structure the config?

We introduce a new preferences object to the configuration file schema. This holds all optional options (see below). It can be specified on the top level for global preferences and inside every context for context-specific preferences.

The only options not included in the preferences are:

• (Global) active-context
• (Context) name
• (Context) token

Namespaced option keys are set in a nested object inside of preferences.

Preferences configured globally apply to all contexts. If a context sets a specific option in the context preferences, this override the global value.

Users can “unset / ignore” global preferences on a context-level by assigning the zero value for its data-type (””, 0, 0.0). Unsetting booleans is not possible. This allows to retreive the next available option value by precedence, skipping the global preferences.

The keys used in the configuration file equal the option keys, but we replace any - with _ to match the existing keys we had before the design doc.

Example of a valid config structure:

active_context = "my-context" # Currently active context (optional)

# Any optional option is managed under preferences.
[preferences]
location = "fsn1" # Global default location preference

[preferences.sort]
certificate = "id:asc"

[[contexts]]
name = "my-context"
token = "def"

[contexts.preferences] # Confusingly, this belongs to the "my-context" context in TOML
location = "hel1" # Override for the global "location" option

# Examples of namespaced preferences:
[contexts.preferences.sort]
server = "created:asc"
load_balancer = "id:desc"

[contexts.preferences.columns]
server = "name,id,age"
load_balancer = "$defaults,age,!id" # Some options to expand the columns syntax in the future

How do we persist the configuration file?

🗣️ This section only describes the current state. We do not plan an making any changes, besides adding a new `--config` flag.

The configuration file is encoded as TOML. By default we look in these paths for the file:

• Default config path:
  • Unix: ~/.config/hcloud/cli.toml
  • Windows: %APPDATA%/hcloud/cli.toml

The config path can be overridden using the HCLOUD_CONFIG env variable or the --config global flag.

When the user makes any changes, we re-write the full file. We do not take any precautions to persist user formatting or comments. This makes it way easier for us to write out any changes back into the file.

Environment Variables

We want to allow users to set all options through environment variables.

Our environment variables always start with HCLOUD_ and then use the option key in uppercase and with - and . replaced with _.

• Examples:
  • token → HCLOUD_TOKEN
  • default-ssh-keys → HCLOUD_DEFAULT_SSH_KEYS
  • sort.server → HCLOUD_SORT_SERVER

Flags

We want to allow users to set options through flags.

The flag names are the option key with . replaced with -.

• Examples:
  • token → --token (only example, flag not present in CLI)
  • quiet → --quiet
  • namespace.foo → --namespace-foo

We will not add global flags for options that already have flags in sub-commands (See above).

Precedence

The value for an option can be set in many different places. We apply the following precedence to decide which value will be used.

1. Interactive Mode (not yet implemented)
2. Flags
3. Environment Variables
4. Configuration File
   1. Context-specific
   2. Global
5. CLI Defaults

The order matches how “far” the user is from where the option was set, with “further” sources having a lower priority.

⇒ The user is very near and interacts directly with the interactive mode or flags. Environment variables have been set before running the command. The configuration file is in another directory altogether and mostly hidden from the user. CLI Defaults are statically set by us in code, the user has no direct influence on this.

Backwards Compatible Options

Some options currently do not have consistent flags/config file keys/environment variables. We do not want to break the existing usages. This section documents any options that deviate from the ideal naming scheme.

• Option Key: token
  • Env: HCLOUD_TOKEN (Already exists)
  • Flag: Not present because of security concerns ⚠️
  • Config: token (Already exists)
• Option Key: context
  • Env: HCLOUD_CONTEXT (Already exists)
  • Flag: --context
  • Config: active_context (Already exists) ⚠️
• Option Key: config
  • Env: HCLOUD_CONFIG (Already exists)
  • Flag: --config
  • Config: We do not plan to add this, as it is redundant and unclear what it would mean. ⚠️

Documentation

We plan to add a new sub-command hcloud help config. In there we will:

• Explain what options are
• Explain exceptions to schema (--token flag not present, HCLOUD_CONTEXT vs active_context etc.)
• Explain where users can set the options
• Explain the renaming of configs between env variables, flags and the config file
• List all available options with a short explanation on their purpose.

Release Strategy

We will cut a normal CLI Release once this is merged and available for users.

Alternatives

Alternative configuration commands: User edits config file themselves

→ Benefits: Less work for us

→ Why we decided against it: Bad UX, no validation / feedback for the user.

Alternative configuration commands: Specialized sub-commands for every option

• Example: hcloud context default-ssh-keys add <ssh-key>

→ Benefits: Intuitive command layout, easy customization

→ Why we decided against it: Lots of clutter, would need a lot of documentation, more effort to maintain

Alternative handling of flags: All flags are handled by viper

• Every flag becomes an option
  • Example: hcloud server create --location flag becomes server.create.location option
• Options can be defined in the config, via environment variables or via flags
• Because of the hierarchy, usage stays consistent, with the difference that you can now define defaults for flags using the configuration file or environment variables

→ Benefits: Only one source of truth (Viper) and every flag is configurable

→ Why we decided against it: Requires major rewrite, very complex implementation for little benefit, no shared options between commands

Alternative config file schema: Namespaced option keys saved flat

→ Benefits:

• Easier implementation

→ Why we decided against it

• Nested objects provide better structure for configuration file

Alternative config command name: hcloud context config

→ Benefits: Makes apparent that configuration happens on a context level

→ Why we decided against it: Bad UX