From 28fa9169a5bca9b4b0721e978f174aa3494a886b Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Wed, 2 Aug 2023 13:00:39 +0100 Subject: [PATCH 1/8] Add migration steps for CL to OCL Signed-off-by: Merel Theisen --- docs/source/configuration/migration.md | 56 ++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 docs/source/configuration/migration.md diff --git a/docs/source/configuration/migration.md b/docs/source/configuration/migration.md new file mode 100644 index 0000000000..2e18ec7f68 --- /dev/null +++ b/docs/source/configuration/migration.md @@ -0,0 +1,56 @@ +## Migration guide for config loaders +The `ConfigLoader` and `TemplatedConfigLoader` classes are now deprecated and scheduled for removal in Kedro `0.19.0`. To ensure smooth transitions for users, we strongly recommend adopting the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) promptly. +This migration guide outlines the primary distinctions between the old loaders and the `OmegaConfigLoader`, providing step-by-step instructions on updating your code base to utilise the new class effectively. + +### [`ConfigLoader`](/kedro.config.ConfigLoader) to [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) + +### 1. Install the Required Library +The [`OmegaConfigLoader`](advanced_configuration.md#omegaconfigloader) was introduced in Kedro `0.18.5` and is based on [OmegaConf](https://omegaconf.readthedocs.io/). In order to use it you need to ensure you have both a version of Kedro of `0.18.5` or above and `omegaconf` installed. +You can install both using `pip`: + +```bash +pip install kedro==0.18.5 +``` +This would be the minimum required Kedro version which includes `omegaconf` as dependency. +Or you can run: +```bash +pip install -U kedro +``` + +This command installs the most recent version of Kedro which also includes `omegaconf` as dependency. + +### 2. Import Statements +Replace the import statement for `ConfigLoader` with the one for `OmegaConfigLoader`: + +```python +# Before: +from kedro.config import ConfigLoader + +# After: +from kedro.config import OmegaConfigLoader +``` + +### 3. File Format Support +`OmegaConfigLoader` supports only `yaml` and `json` file formats. Make sure that all your configuration files are in one of these formats. If you were using other formats with `ConfigLoader`, convert them to `yaml` or `json`. + +### 4. Load Configuration +The method to load the configuration using `OmegaConfigLoader` is slightly updated. The `ConfigLoader` allowed users to access configuration through the `.get()` method, which required patterns as argument. +The `OmegaConfigLoader` requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. + +```python +# Before: +conf_path = str(project_path / settings.CONF_SOURCE) +conf_loader = ConfigLoader(conf_source=conf_path, env="local") +catalog = conf_loader.get("catalog*") + +# After: +conf_path = str(project_path / settings.CONF_SOURCE) +config_loader = OmegaConfigLoader(conf_source=conf_path, env="local") +catalog = config_loader["catalog"] +``` + +In this example, `"catalog"` is the key to the default catalog patterns specified in the `OmegaConfigLoader` class. + +### 5. Exception Handling +* `OmegaConfigLoader` throws a `MissingConfigException` when configuration paths don't exist, rather than the `ValueError` used in `ConfigLoader`. +* In `OmegaConfigLoader`, if there is bad syntax in your configuration files, it will trigger a `ParserError` instead of a `BadConfigException` used in `ConfigLoader`. From 798f60886813b93869f4e13862adb0e6daa9af09 Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 2 Aug 2023 16:50:44 +0100 Subject: [PATCH 2/8] Add new migration guide to index Signed-off-by: Jo Stichbury --- .../configuration/{migration.md => config_loader_migration.md} | 2 +- docs/source/configuration/index.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) rename docs/source/configuration/{migration.md => config_loader_migration.md} (98%) diff --git a/docs/source/configuration/migration.md b/docs/source/configuration/config_loader_migration.md similarity index 98% rename from docs/source/configuration/migration.md rename to docs/source/configuration/config_loader_migration.md index 2e18ec7f68..9d11928c49 100644 --- a/docs/source/configuration/migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -1,4 +1,4 @@ -## Migration guide for config loaders +# Migration guide for config loaders The `ConfigLoader` and `TemplatedConfigLoader` classes are now deprecated and scheduled for removal in Kedro `0.19.0`. To ensure smooth transitions for users, we strongly recommend adopting the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) promptly. This migration guide outlines the primary distinctions between the old loaders and the `OmegaConfigLoader`, providing step-by-step instructions on updating your code base to utilise the new class effectively. diff --git a/docs/source/configuration/index.md b/docs/source/configuration/index.md index 3f554e1e91..291a4fbf65 100644 --- a/docs/source/configuration/index.md +++ b/docs/source/configuration/index.md @@ -6,5 +6,6 @@ configuration_basics credentials parameters +config_loader_migration advanced_configuration ``` From c2d3c8e9f1b912a1413131e7729cd3058ea57b72 Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Thu, 3 Aug 2023 16:54:15 +0100 Subject: [PATCH 3/8] Address review comments Signed-off-by: Merel Theisen --- .../configuration/config_loader_migration.md | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/source/configuration/config_loader_migration.md b/docs/source/configuration/config_loader_migration.md index 9d11928c49..01d6eb917f 100644 --- a/docs/source/configuration/config_loader_migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -1,8 +1,8 @@ # Migration guide for config loaders -The `ConfigLoader` and `TemplatedConfigLoader` classes are now deprecated and scheduled for removal in Kedro `0.19.0`. To ensure smooth transitions for users, we strongly recommend adopting the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) promptly. +The `ConfigLoader` and `TemplatedConfigLoader` classes have been deprecated since Kedro `0.18.12` and will be removed in Kedro `0.19.0`. To ensure a smooth transition, we strongly recommend you adopt the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) as soon as possible. This migration guide outlines the primary distinctions between the old loaders and the `OmegaConfigLoader`, providing step-by-step instructions on updating your code base to utilise the new class effectively. -### [`ConfigLoader`](/kedro.config.ConfigLoader) to [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) +## [`ConfigLoader`](/kedro.config.ConfigLoader) to [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) ### 1. Install the Required Library The [`OmegaConfigLoader`](advanced_configuration.md#omegaconfigloader) was introduced in Kedro `0.18.5` and is based on [OmegaConf](https://omegaconf.readthedocs.io/). In order to use it you need to ensure you have both a version of Kedro of `0.18.5` or above and `omegaconf` installed. @@ -19,7 +19,16 @@ pip install -U kedro This command installs the most recent version of Kedro which also includes `omegaconf` as dependency. -### 2. Import Statements +### 2. Use the `OmegaConfigLoader` +To use `OmegaConfigLoader` in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): + +```python +from kedro.config import OmegaConfigLoader # new import + +CONFIG_LOADER_CLASS = OmegaConfigLoader +``` + +### 3. Import Statements Replace the import statement for `ConfigLoader` with the one for `OmegaConfigLoader`: ```python @@ -30,12 +39,12 @@ from kedro.config import ConfigLoader from kedro.config import OmegaConfigLoader ``` -### 3. File Format Support +### 4. File Format Support `OmegaConfigLoader` supports only `yaml` and `json` file formats. Make sure that all your configuration files are in one of these formats. If you were using other formats with `ConfigLoader`, convert them to `yaml` or `json`. -### 4. Load Configuration -The method to load the configuration using `OmegaConfigLoader` is slightly updated. The `ConfigLoader` allowed users to access configuration through the `.get()` method, which required patterns as argument. -The `OmegaConfigLoader` requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. +### 5. Load Configuration +The method to load the configuration using `OmegaConfigLoader` differs slightly from that used by `ConfigLoader`, which allowed users to access configuration through the `.get()` method and required patterns as argument. +When you migrate to use `OmegaConfigLoader`it requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. ```python # Before: @@ -51,6 +60,6 @@ catalog = config_loader["catalog"] In this example, `"catalog"` is the key to the default catalog patterns specified in the `OmegaConfigLoader` class. -### 5. Exception Handling +### 6. Exception Handling * `OmegaConfigLoader` throws a `MissingConfigException` when configuration paths don't exist, rather than the `ValueError` used in `ConfigLoader`. * In `OmegaConfigLoader`, if there is bad syntax in your configuration files, it will trigger a `ParserError` instead of a `BadConfigException` used in `ConfigLoader`. From bd624f96dd232dce9e1227358529762510fdd8f1 Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Fri, 4 Aug 2023 11:01:59 +0100 Subject: [PATCH 4/8] Try diff highlighting Signed-off-by: Merel Theisen --- .../configuration/config_loader_migration.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/source/configuration/config_loader_migration.md b/docs/source/configuration/config_loader_migration.md index 01d6eb917f..68a7cfd307 100644 --- a/docs/source/configuration/config_loader_migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -22,21 +22,19 @@ This command installs the most recent version of Kedro which also includes `omeg ### 2. Use the `OmegaConfigLoader` To use `OmegaConfigLoader` in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): -```python -from kedro.config import OmegaConfigLoader # new import +```diff ++ from kedro.config import OmegaConfigLoader # new import -CONFIG_LOADER_CLASS = OmegaConfigLoader ++ CONFIG_LOADER_CLASS = OmegaConfigLoader ``` ### 3. Import Statements Replace the import statement for `ConfigLoader` with the one for `OmegaConfigLoader`: -```python -# Before: -from kedro.config import ConfigLoader +```diff +- from kedro.config import ConfigLoader -# After: -from kedro.config import OmegaConfigLoader ++ from kedro.config import OmegaConfigLoader ``` ### 4. File Format Support From 1e8b76c05e1d0b582381a5f6ba5e12f1e9b36c99 Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Fri, 4 Aug 2023 11:08:06 +0100 Subject: [PATCH 5/8] Add diff highlight to all examples Signed-off-by: Merel Theisen --- .../configuration/config_loader_migration.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/source/configuration/config_loader_migration.md b/docs/source/configuration/config_loader_migration.md index 68a7cfd307..1cd83feb2e 100644 --- a/docs/source/configuration/config_loader_migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -44,16 +44,14 @@ Replace the import statement for `ConfigLoader` with the one for `OmegaConfigLoa The method to load the configuration using `OmegaConfigLoader` differs slightly from that used by `ConfigLoader`, which allowed users to access configuration through the `.get()` method and required patterns as argument. When you migrate to use `OmegaConfigLoader`it requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. -```python -# Before: -conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = ConfigLoader(conf_source=conf_path, env="local") -catalog = conf_loader.get("catalog*") - -# After: -conf_path = str(project_path / settings.CONF_SOURCE) -config_loader = OmegaConfigLoader(conf_source=conf_path, env="local") -catalog = config_loader["catalog"] +```diff +- conf_path = str(project_path / settings.CONF_SOURCE) +- conf_loader = ConfigLoader(conf_source=conf_path, env="local") +- catalog = conf_loader.get("catalog*") + ++ conf_path = str(project_path / settings.CONF_SOURCE) ++ config_loader = OmegaConfigLoader(conf_source=conf_path, env="local") ++ catalog = config_loader["catalog"] ``` In this example, `"catalog"` is the key to the default catalog patterns specified in the `OmegaConfigLoader` class. From de9c3b2daddf28cb9dd5c533c0a4bb3d3dd9530d Mon Sep 17 00:00:00 2001 From: Merel Theisen <49397448+merelcht@users.noreply.github.com> Date: Tue, 8 Aug 2023 10:52:37 +0100 Subject: [PATCH 6/8] Apply suggestions from code review Co-authored-by: Jo Stichbury --- docs/source/configuration/config_loader_migration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/configuration/config_loader_migration.md b/docs/source/configuration/config_loader_migration.md index 1cd83feb2e..876920b5b2 100644 --- a/docs/source/configuration/config_loader_migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -11,13 +11,13 @@ You can install both using `pip`: ```bash pip install kedro==0.18.5 ``` -This would be the minimum required Kedro version which includes `omegaconf` as dependency. +This would be the minimum required Kedro version which includes `omegaconf` as a dependency. Or you can run: ```bash pip install -U kedro ``` -This command installs the most recent version of Kedro which also includes `omegaconf` as dependency. +This command installs the most recent version of Kedro which also includes `omegaconf` as a dependency. ### 2. Use the `OmegaConfigLoader` To use `OmegaConfigLoader` in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): @@ -38,11 +38,11 @@ Replace the import statement for `ConfigLoader` with the one for `OmegaConfigLoa ``` ### 4. File Format Support -`OmegaConfigLoader` supports only `yaml` and `json` file formats. Make sure that all your configuration files are in one of these formats. If you were using other formats with `ConfigLoader`, convert them to `yaml` or `json`. +`OmegaConfigLoader` supports only `yaml` and `json` file formats. Make sure that all your configuration files are in one of these formats. If you previously used other formats with `ConfigLoader`, convert them to `yaml` or `json`. ### 5. Load Configuration The method to load the configuration using `OmegaConfigLoader` differs slightly from that used by `ConfigLoader`, which allowed users to access configuration through the `.get()` method and required patterns as argument. -When you migrate to use `OmegaConfigLoader`it requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. +When you migrate to use `OmegaConfigLoader` it requires you to fetch configuration through a configuration key that points to [configuration patterns specified in the loader class](configuration_basics.md#configuration-patterns) or [provided in the `CONFIG_LOADER_ARGS`](advanced_configuration.md#how-to-change-which-configuration-files-are-loaded) in `settings.py`. ```diff - conf_path = str(project_path / settings.CONF_SOURCE) From f418cdc65fbdc143970aadf00850a8ba6664c994 Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Tue, 8 Aug 2023 10:55:30 +0100 Subject: [PATCH 7/8] Address review comment Signed-off-by: Merel Theisen --- docs/source/configuration/config_loader_migration.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/configuration/config_loader_migration.md b/docs/source/configuration/config_loader_migration.md index 876920b5b2..f27b244afe 100644 --- a/docs/source/configuration/config_loader_migration.md +++ b/docs/source/configuration/config_loader_migration.md @@ -57,5 +57,6 @@ When you migrate to use `OmegaConfigLoader` it requires you to fetch configurat In this example, `"catalog"` is the key to the default catalog patterns specified in the `OmegaConfigLoader` class. ### 6. Exception Handling +For error and exception handling, most errors are the same. Those you need to be aware of that are different between the original `ConfigLoader` and `OmegaConfigLoader` are as follows: * `OmegaConfigLoader` throws a `MissingConfigException` when configuration paths don't exist, rather than the `ValueError` used in `ConfigLoader`. * In `OmegaConfigLoader`, if there is bad syntax in your configuration files, it will trigger a `ParserError` instead of a `BadConfigException` used in `ConfigLoader`. From 5dfe57bcfc17dc3127a97b69771b78a55539dacb Mon Sep 17 00:00:00 2001 From: Merel Theisen Date: Tue, 8 Aug 2023 11:01:25 +0100 Subject: [PATCH 8/8] Update release notes Signed-off-by: Merel Theisen --- RELEASE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/RELEASE.md b/RELEASE.md index 9d7d712eb2..61cc91f985 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -17,6 +17,7 @@ * Updated `kedro pipeline create` to use new `/conf` file structure. ## Documentation changes +* Added migration guide from the `ConfigLoader` to the `OmegaConfigLoader`. The `ConfigLoader` is deprecated and will be removed in the `0.19.0` release. ## Breaking changes to the API