Draft - Resolves configuration earlier & separate runtimes_params and meta_params

[noklam]

Signed-off-by: Nok Chan [email protected]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

1. Resolve runtime_params earlier
2. Fix problem when len(aggregate_config) == 1

ds_name='example_iris_data' ds_config={'type': 'pandas.CSVDataSet', 'filepath': '/Users/Nok_Lam_Chan/dev/test/omega-nest-param/data/01_raw/iris.csv'}

│ /Users/Nok_Lam_Chan/GitHub/kedro/kedro/io/data_catalog.py:272 in from_config                     │
│                                                                                                  │
│   269 │   │                                                                                      │
│   270 │   │   layers = defaultdict(set)  # type: Dict[str, Set[str]]                             │
│   271 │   │   for ds_name, ds_config in catalog.items():                                         │
│ ❱ 272 │   │   │   ds_layer = ds_config.pop("layer", None)                                        │
│   273 │   │   │   if ds_layer is not None:                                                       │
│   274 │   │   │   │   layers[ds_layer].add(ds_name)  

4. Separate runtime_params & meta_params
5. Bonus: It may also simplify the syntax - since meta_params will not be pass to catalog and avoid the validation problem.

Development notes

Challenge 1 - Parameters resolved too late
Once we merge parameters in ConfigLoader, all runtime_params are merged. This is fine for parameters, but when it's done for catalog -

Challenge 2 - Works for parameters but not for other configs
We get invalid entries and we don't really want it. Same goes for logging. This wasn't an issue in #2467 because of an accident.

Currently this PR takes a simple solution - it is only enable for parameters and nothing else. This is actually the same problem
of #2175, but was hidden because of the above logic.

kedro/kedro/config/omegaconf_config.py

Lines 276 to 278 in 7dcb60c

	if len(aggregate_config) == 1:
	return list(aggregate_config)[0]
	return dict(OmegaConf.merge(*aggregate_config))

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes