Append new fields to primary YAML config by default

[addisonklinke]

Say I have a YAML defining the main config

# configs/config.yaml
foo: bar
baz: 1

And I want the app to add a new, top-level field to the composed config (as the default behavior). So far I have

# app.py
from dataclasses import dataclass
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import OmegaConf


@dataclass
class Extras:
    hello: str = 'world'


cs = ConfigStore.instance()
cs.store(group='extras', name='example', package='_global_', node=Extras)


@hydra.main(config_name='config', config_path='configs')
def main(cfg):
    resolved_cfg = OmegaConf.to_container(cfg, resolve=True, enum_to_str=True)
    print(OmegaConf.to_yaml(resolved_cfg))


if __name__ == '__main__':
    main()

This accomplishes the goal if I add an override for the extras config group

$ python app.py +extras=example
foo: bar
baz: 1
hello: world

But without the override, the hello: world field is missing from the composed config. How can I alter the defaults list of config.yaml from app.py so that the +extras=example is not necessary to produce the above output? I've tried adding a defaults list to the Extras dataclass, but cannot find an option that works

Approach 1: specified the group and ran into the relative path error explained by @Jasha10 here

{'extras': 'example'}
In 'extras/example': Could not find 'extras/extras/example'

Approach 2: so I try a leading slash to make the group absolute

{'/extras': 'example'}
RecursionError: maximum recursion depth exceeded in comparison

Approach 3: add specification for the global package, same error

{'/extras@_global_': 'example'}
RecursionError: maximum recursion depth exceeded in comparison

Am I just running into a fundamental limitation - namely that the only way to adjust my defaults is editing the YAML directly, or is there a more clever solution to this that I'm missing? If all else fails, I suppose I can always hack sys.argv to add the override string myself as discussed in this comment on reproducing old experiments. Coincidentally, that's my underlying motivation for this use case - I want to re-use the majority of an old config while appending new fields to adapt it for a slightly different app

[Jasha10]

I think passing an override will be the easiest way go, which means hacking sys.argv or using the the overrides keyword to compose.

Another idea is to use a different config as top-level and pull in configs/config.yaml as a subconfig:

# app2.py
from dataclasses import dataclass
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import OmegaConf

@dataclass
class Extras:
    hello: str = "world"

cs = ConfigStore.instance()
cs.store(group="extras", name="example", package="_global_", node=Extras)

top_level_config = {
    "defaults": [
        "config",  # this pulls in `configs/config.yaml`
        {"extras": "example"},
    ]
}
cs.store(name="top_level_config", node=top_level_config)

@hydra.main(config_name="top_level_config", config_path="configs", version_base=None)
def main(cfg):
    resolved_cfg = OmegaConf.to_container(cfg, resolve=True, enum_to_str=True)
    print(OmegaConf.to_yaml(resolved_cfg))

if __name__ == "__main__":
    main()

$ python app2.py
foo: bar
baz: 1
hello: world

Calling cs.store on top_level_config here is equivalent to creating a file configs/top_level_config.yaml with the following contents:

defaults:
  - config
  - extras: example

I think there might some edge cases where the override method will work more reliably than the top_level_config method, such as if configs/config.yaml defines a hydra.sweeper.params key (which I believe only works if it's defined in the top-level config).