Improve OmegaConfigLoader performance

[ravi-kumar-pilla]

Description

Resolves #4322

Development notes

• As mentioned by @noklam here, this PR makes a quick fix for _get_globals_value and _get_runtime_value by not re-creating _globals and runtime_params for each reference in the catalog
• As mentioned by @matthias here, separated the config merge and underscore filter which improved the performance as well
• This is a minor improvement to the performance of OCL. The bottleneck still remains the use of
  OmegaConf.load
  OmegaConf.to_container
  OmegaConf.merge

Test Data

# globals.yml

compression_type: '${random_choice: gzip, bz2, xz, None}'
random_seed: 42
separator: "${random_choice: ,, \t, |}"

# catalog.yml with variable number of entries

interpolated_dataset_3610:
  filepath: data/interpolated_dataset_3610.csv
  load_args:
    encoding: '${random_choice: utf-8, iso-8859-1, utf-16}'
    sep: ${globals:separator}
  save_args:
    compression: ${globals:compression_type}
    index: false
  type: pandas.CSVDataSet

Before:

---

After:

---

Observation:
There is a slight improvement in the performance as seen above

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[astrojuanlu]

34 % reduction, not bad 👏🏼

[ElenaKhaustova]

Implementation looks good to me, thanks @ravi-kumar-pilla!

[noklam]

I have tested the PR and have the same findings, the bottleneck of resolver has been removed with caching mechanism implemented in this PR, but there are bottlenecks remains from OmegaConf.

I have reduce the issue into a single large config with no resolver involved.
File: catalog_datasets_with_variables.yml.zip

It can be benchmark with pyinstrument with a simple script like

from omegaconf import OmegaConf
OmegaConf.load(path)

or

%%time
OmegaConf.load(path)

This takes a significant time to run compare to yaml.safe_load, which is instant.

[noklam]

Approved with comments to reference in the future. Also please update the release notes.