Set configuration options via environment variables

[DominicOram]

As a user trying to use the client with stomp it is frustrating that I have to provide the config file every time I call the CLI. It would be better if I could specify this as an environment variable so that I do not need to keep specifying it. We could do either:

• export BLUEAPI.config.file=config_file - this is the same sort of file we would currently use
• export BLUEAPI.config.api.host=my_host - we can then split out all the config fields

[stan-dot]

@DominicOram what is your preference?

what if we have ~/.blueapi/config.yaml as a default location? it feels unix-like

[stan-dot]

@callumforrester what was the reasoning to require it always in the first place?

[DominicOram]

No preference, as long as it's well documented I'm not too fussed

[callumforrester]

@stan-dot the configuration file is not always required per-se, but only when you want configuration that's different to the defaults. The intention was always to provide other ways of overriding configuration than a file, which is a bit clunky for a single value. That is what this ticket is for.

For that reason, I have a preference for the second of @DominicOram's options. It's relatively common to only want to override one value, for example export BLUEAPI.config.stomp.host=localhost would automatically turn on the message bus for local dev and make #615 easier. It seems like overkill to need a whole file for this. Let alone to need to know a default location where you need to put it to make the magic work. N.b. this is where I struggle with databroker sometimes (see docs), it's nice to be able to do something simple just from the terminal.

Eventually I would like to see blueapi use the same configuration override order that @tpoliaw has introduced to GDA:

1. Default values, unless overridden by...
2. File, unless overridden by...
3. Environment variables, unless overridden by...
4. Values passed in via a CLI argument

[callumforrester]

So the long and short of it.

Acceptance Criteria

1. If I run export BLUEAPI.config.<some value>, the default configuration for that value (found in blueapi/src/config.py is overridden by <some value>
2. If I run the above and the value is already specified in the configuration file, the environment variable still overrides
3. The docs explain this
4. Tests cover this
5. The temporary parsing of environment variables for just the message bus password (see init container cannot be run when mounting configuration values from envVars without also mounting those envVars to the init container #620) is removed, this mechanism would replace it

[callumforrester]

This issue also relates to/could be done at the same time as #532

[stan-dot]

Eventually I would like to see blueapi use the same configuration override order that @tpoliaw has introduced to GDA:
Default values, unless overridden by...
File, unless overridden by...
Environment variables, unless overridden by...
Values passed in via a CLI argument

What is the rationale for this? it's more functionality exposed, and users might come to use all 4 in various combinations hard to debug.

the Zen of Python says:

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

passing a file path is bulky, a long override in the cli also feels bulky

Let alone to need to know a default location where you need to put it to make the magic work.

it's literally how most unix tools keep config, in the home directory in a hidden file, or in .config. It is a common pattern.

you say that a 'whole file is overkill', implying a spectrum of 'sizes' from a cli argument to file, where an etcd cache could be postulated as an extrapolation. I don't think the size is the relevant criterion here, but what is easy to work with.

Files are easier to debug imo than env variables. One potential difference might be if the config has any secret fields, then in deployment some extra setup would be needed.

[callumforrester]

passing a file path is bulky, a long override in the cli also feels bulky

Indeed, but the following feels like the least bulky solution:

export BLUEAPI.config.stomp.host=localhost
blueapi serve

You could easily just have this in a quickstart tutorial in the docs and it would take someone 5 seconds, as opposed to:

1. Create a file in /some/path
2. Add the following contents:

stomp:
  host: localhost

3. Run blueapi serve
4. Don't forget you might be in a devcontainer...

[stan-dot]

this could all be addressed, but I concede for time savings now

[stan-dot]

@callumforrester should the issue just be to do the whole override system with 4 layers?

[callumforrester]

No, just environment variables (see acceptance criteria above)

[stan-dot]

this issue is purely UX/DX, we could make it full from the first time.

[stan-dot]

have to provide the config file every time I call the CLI

@DominicOram how exactly are you calling the CLI? I realized I'm not familiar how does mx-hyperion use blueapi? or is it for manual setup? there is the REST api that I'd expect to be what most of the users interact with

[DominicOram]

have to provide the config file every time I call the CLI

@DominicOram how exactly are you calling the CLI? I realized I'm not familiar how does mx-hyperion use blueapi? or is it for manual setup? there is the REST api that I'd expect to be what most of the users interact with

This is just for manual setup and testing. We don't use it regularly in production so it's not a high priority for us

[stan-dot]

and do you run it locally on beamline workstations or DH workstations or in the cluster?

[stan-dot]

following @DiamondJoseph 's suggestion I will look into the pydantic setup taking this https://stackoverflow.com/questions/75709741/pydantic-nested-setting-objects-load-env-variables-from-file/75709846#75709846

into account

[DominicOram]

and do you run it locally on beamline workstations or DH workstations or in the cluster?

Beamline workstations and DH workstations

[stan-dot]

@callumforrester if I am to implement this, I must know some things:

• why wasn't the BaseSettings class used in the first place when doing this with Pydantic?
• are there any blockers to migrating this now?
• do you have any strong opinions at this stage - priori to the draft implementation?
  https://docs.pydantic.dev/latest/concepts/pydantic_settings/#environment-variable-names

[callumforrester]

@stan-dot I did not know the BaseSettings class can parse config files as well as environment variables. It is possible that it couldn't at the time I implemented configuration (with pydantic 1) or I could have missed something. I agree this looks like a promising way to go!

I have no strong opinions with the user-facing side as long as @DominicOram is happy.

I am concerned that turning this into a big refactor of the existing config significantly increases the scope of this issue. Perhaps we need to break it down. For example:

• Refactor existing config parsing to use pydantic settings
• Enable override via environment variables with whatever prefix/delimiter we settle on

Not quite sure if that's a sensible order to do things from reading the docs, thoughts welcome.

[DiamondJoseph]

First refactoring to use BaseSettings, then documenting how to pass overrides seems a sensible direction to me.

Looks like Pydantic's default order is different from our expectations, but I think we take their order.

[stan-dot]

thanks for the response @callumforrester

It is possible that it couldn't at the time I implemented configuration
that's what I thought

Yes, making the 'migrate to basesettings' as a separate issue is the best, and addressing this only after that is complete.

[stan-dot]

unassigned as the basesettings PR was closed due to use case mismatch

[callumforrester]

I would suggest we go with @DominicOram's other suggestion

export BLUEAPI.config.file=config_file - this is the same sort of file we would currently use

Since the original plan has entailed a lot of unexpected complexity