OmegaConfigLoader requires consistent config for each env

[KrzysztofDobosz]

Description

OmegaConfigLoader checks for completeness config for each env before updating base config with values from local. This prevents defining global values for data catalog in base config only.

Context

I was trying to use templating with Omega in v 0.18.10. I have sql queries which form spark.SparkJDBCDataSet in base and pandas.SQLQueryDataSet in local. My goal was to be able to define the query just in one place (conf/base/catalog_globals.yml).

Steps to Reproduce

1. Define a global value in conf/base/catalog_globals.yml
2. Use this value in dataset in both base and local config envs.

Expected Result

Config from base is updated with values from local, and then templated values are resolved.

Actual Result

This exception is thrown, unless I define globals also in local env (which I was trying to avoid).

InterpolationKeyError: Interpolation key '***' not found

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.18.10
• Python version used (python -V): 3.9
• Operating system and version: Windows 10

[astrojuanlu]

Hi @KrzysztofDobosz, thanks for opening this issue. Could you share a YAML snippet of what do you mean by:

This prevents defining global values for data catalog in base config only.

[KrzysztofDobosz]

Hi @astrojuanlu. Sure. I have something very similar in the config files:
base/catalog_globals.yml

_query:
  "SELECT * FROM table"

base/catalog.yml

query_dataset:
  type: spark.SparkJDBCDataSet
  layer: raw
  url: jdbc:sqlserver://(...)
  table: "(${_query}) alias"

local/catalog_globals.yml is empty
local/catalog.yml

query_dataset:
  type: pandas.SQLQueryDataSet
  layer: raw
  sql: ${_query}

Unless I copy _query to local/catalog_globals.yml, I see InterpolationKeyError.

Actually, while trying to reproduce mentioned error, I noticed that if I define template values in local/catalog_globals.yml with different values than in base env, the values from base env are used. That's probably because:

Templated values will neither trigger a key duplication error nor appear in the resulting configuration dictionary.

In my project SQL queries are identical in all envs, but some other template variables aren't. Can I use them somehow with OmegaConfigLoader?

[ankatiyar]

Thanks for reporting this @KrzysztofDobosz! I've been able to reproduce it. Working on a fix.

[ankatiyar]

Hey @KrzysztofDobosz, turns out this is expected behaviour for now. We're still working on adding globals functionality for OmegaConfigLoader #2175. Closing this issue but feel free to the discussion on the globals issue! :)