diff --git a/RELEASE.md b/RELEASE.md index 783d837d18..3b2cb8249f 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -13,6 +13,7 @@ ## Major features and improvements * `kedro run --params` now updates interpolated parameters correctly when using `OmegaConfigLoader`. * Added `metadata` attribute to `kedro.io` datasets. This is ignored by Kedro, but may be consumed by users or external plugins. +* Added `kedro.logging.RichHandler`. This replaces the default `rich.logging.RichHandler` and is more flexible, user can turn off the `rich` traceback if needed. ## Bug fixes and other changes * `OmegaConfigLoader` will return a `dict` instead of `DictConfig`. @@ -141,12 +142,12 @@ Many thanks to the following Kedroids for contributing PRs to this release: * You can configure config file patterns through `settings.py` without creating a custom config loader. * Added the following new datasets: -| Type | Description | Location | -| ------------------------------------ | -------------------------------------------------------------------------- | ----------------------------- | -| `svmlight.SVMLightDataSet` | Work with svmlight/libsvm files using scikit-learn library | `kedro.extras.datasets.svmlight` | -| `video.VideoDataSet` | Read and write video files from a filesystem | `kedro.extras.datasets.video` | -| `video.video_dataset.SequenceVideo` | Create a video object from an iterable sequence to use with `VideoDataSet` | `kedro.extras.datasets.video` | -| `video.video_dataset.GeneratorVideo` | Create a video object from a generator to use with `VideoDataSet` | `kedro.extras.datasets.video` | +| Type | Description | Location | +| ------------------------------------ | -------------------------------------------------------------------------- | -------------------------------- | +| `svmlight.SVMLightDataSet` | Work with svmlight/libsvm files using scikit-learn library | `kedro.extras.datasets.svmlight` | +| `video.VideoDataSet` | Read and write video files from a filesystem | `kedro.extras.datasets.video` | +| `video.video_dataset.SequenceVideo` | Create a video object from an iterable sequence to use with `VideoDataSet` | `kedro.extras.datasets.video` | +| `video.video_dataset.GeneratorVideo` | Create a video object from a generator to use with `VideoDataSet` | `kedro.extras.datasets.video` | * Implemented support for a functional definition of schema in `dask.ParquetDataSet` to work with the `dask.to_parquet` API. ## Bug fixes and other changes diff --git a/docs/source/conf.py b/docs/source/conf.py index 58175c2c8b..7aa16b79e5 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -191,7 +191,7 @@ here = Path(__file__).parent.absolute() html_logo = str(here / "kedro_logo.svg") -# Theme options are theme-specific and customize the look and feel of a theme +# Theme options are theme-specific and customise the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # diff --git a/docs/source/kedro.logging.RichHandler.rst b/docs/source/kedro.logging.RichHandler.rst new file mode 100644 index 0000000000..b14b9e91aa --- /dev/null +++ b/docs/source/kedro.logging.RichHandler.rst @@ -0,0 +1,6 @@ +kedro.logging.RichHandler +========================= + +.. currentmodule:: kedro.logging + +.. autoclass:: RichHandler diff --git a/docs/source/kedro.logging.rst b/docs/source/kedro.logging.rst new file mode 100644 index 0000000000..9476656b50 --- /dev/null +++ b/docs/source/kedro.logging.rst @@ -0,0 +1,20 @@ +kedro.logging +============= + +.. rubric:: Description + +.. automodule:: kedro.logging + + + + + + + + .. rubric:: Classes + + .. autosummary:: + :toctree: + :template: autosummary/class.rst + + RichHandler diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 5260c1296e..93f3332660 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -19,7 +19,7 @@ We now give some common examples of how you might like to change your project's ### Using `KEDRO_LOGGING_CONFIG` environment variable -`KEDRO_LOGGING_CONFIG` is an optional environment variable that you can use to specify the path of your logging configuration file, overriding the default path of `conf/base/logging.yml`. +`KEDRO_LOGGING_CONFIG` is an optional environment variable that you can use to specify the path of your logging configuration file, overriding the default Kedro's `default_logging.yml`. To use this environment variable, set it to the path of your desired logging configuration file before running any Kedro commands. For example, if you have a logging configuration file located at `/path/to/logging.yml`, you can set `KEDRO_LOGGING_CONFIG` as follows: @@ -30,7 +30,7 @@ export KEDRO_LOGGING_CONFIG=/path/to/logging.yml After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path. ```{note} -If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of `conf/base/logging.yml`. +If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` ### Disable file-based logging @@ -43,6 +43,28 @@ Alternatively, if you would like to keep other configuration in `conf/base/loggi + handlers: [console] ``` +### Customise the `rich` Handler + +Kedro's `kedro.extras.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. + +```{note} +If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/base/logging.yml`. +``` + +When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings. + +For instance, you can enable the display of local variables inside `logging.yml` to aid with debugging. + +```yaml +rich: + class: kedro.extras.logging.RichHandler + rich_tracebacks: True + tracebacks_show_locals: True +``` + +A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). + + ### Use plain console logging To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows: diff --git a/features/steps/test_starter/{{ cookiecutter.repo_name }}/conf/base/logging.yml b/features/steps/test_starter/{{ cookiecutter.repo_name }}/conf/base/logging.yml index c8609b142b..d60b7d3592 100644 --- a/features/steps/test_starter/{{ cookiecutter.repo_name }}/conf/base/logging.yml +++ b/features/steps/test_starter/{{ cookiecutter.repo_name }}/conf/base/logging.yml @@ -34,7 +34,11 @@ handlers: delay: True rich: - class: rich.logging.RichHandler + class: kedro.logging.RichHandler + rich_tracebacks: True + # Advance options for customisation. + # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration + # tracebacks_show_locals: False loggers: kedro: diff --git a/kedro/__init__.py b/kedro/__init__.py index 2de9928d4b..0c11f5793f 100644 --- a/kedro/__init__.py +++ b/kedro/__init__.py @@ -4,8 +4,3 @@ """ __version__ = "0.18.8" - - -import logging - -logging.getLogger(__name__).addHandler(logging.NullHandler()) diff --git a/kedro/framework/project/__init__.py b/kedro/framework/project/__init__.py index 741c9fe8d2..0d6946555e 100644 --- a/kedro/framework/project/__init__.py +++ b/kedro/framework/project/__init__.py @@ -7,7 +7,6 @@ import logging.config import operator import os -import sys import traceback import types import warnings @@ -16,10 +15,7 @@ from pathlib import Path from typing import Any -import click import importlib_resources -import rich.pretty -import rich.traceback import yaml from dynaconf import LazySettings from dynaconf.validator import ValidationError, Validator @@ -221,18 +217,6 @@ def __init__(self): ) logging_config = Path(path).read_text(encoding="utf-8") self.configure(yaml.safe_load(logging_config)) - logging.captureWarnings(True) - - # We suppress click here to hide tracebacks related to it conversely, - # kedro is not suppressed to show its tracebacks for easier debugging. - # sys.executable is used to get the kedro executable path to hide the - # top level traceback. - # Rich traceback handling does not work on databricks. Hopefully this will be - # fixed on their side at some point, but until then we disable it. - # See https://github.com/Textualize/rich/issues/2455 - if "DATABRICKS_RUNTIME_VERSION" not in os.environ: - rich.traceback.install(suppress=[click, str(Path(sys.executable).parent)]) - rich.pretty.install() def configure(self, logging_config: dict[str, Any]) -> None: """Configure project logging using ``logging_config`` (e.g. from project diff --git a/kedro/framework/project/default_logging.yml b/kedro/framework/project/default_logging.yml index 5c7b0fd798..87fae8a25c 100644 --- a/kedro/framework/project/default_logging.yml +++ b/kedro/framework/project/default_logging.yml @@ -4,7 +4,11 @@ disable_existing_loggers: False handlers: rich: - class: rich.logging.RichHandler + class: kedro.logging.RichHandler + rich_tracebacks: True + # Advance options for customisation. + # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration + # tracebacks_show_locals: False loggers: kedro: diff --git a/kedro/logging.py b/kedro/logging.py new file mode 100644 index 0000000000..534776c566 --- /dev/null +++ b/kedro/logging.py @@ -0,0 +1,58 @@ +""" +This module contains a logging handler class which produces coloured logs and tracebacks. +""" + +import logging +import os +import sys +from pathlib import Path + +import click +import rich.logging +import rich.pretty +import rich.traceback + + +class RichHandler(rich.logging.RichHandler): + """Identical to rich's logging handler but with a few extra behaviours: + * warnings issued by the `warnings` module are redirected to logging + * pretty printing is enabled on the Python REPL (including IPython and Jupyter) + * all tracebacks are handled by rich when rich_tracebacks=True + * constructor's arguments are mapped and passed to `rich.traceback.install` + + The list of available options of ``RichHandler`` can be found here: + https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler + + The list of available options of `rich.traceback.install` can be found here: + https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + logging.captureWarnings(True) + rich.pretty.install() + + # We suppress click here to hide tracebacks related to it conversely, + # kedro is not suppressed to show its tracebacks for easier debugging. + # sys.executable is used to get the kedro executable path to hide the + # top level traceback. + + traceback_install_kwargs = { + "suppress": [click, str(Path(sys.executable).parent)] + } + + # Mapping arguments from RichHandler's Constructor to rich.traceback.install + prefix = "tracebacks_" + for key, value in kwargs.items(): + if key.startswith(prefix): + key_prefix_removed = key[len(prefix) :] + if key_prefix_removed == "suppress": + traceback_install_kwargs[key_prefix_removed].extend(value) + else: + traceback_install_kwargs[key_prefix_removed] = value + + if self.rich_tracebacks and "DATABRICKS_RUNTIME_VERSION" not in os.environ: + # Rich traceback handling does not work on databricks. Hopefully this will be + # fixed on their side at some point, but until then we disable it. + # See https://github.com/Textualize/rich/issues/2455 + rich.traceback.install(**traceback_install_kwargs) diff --git a/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/base/logging.yml b/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/base/logging.yml index 934e615283..c6a6fc7057 100644 --- a/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/base/logging.yml +++ b/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/base/logging.yml @@ -24,7 +24,11 @@ handlers: delay: True rich: - class: rich.logging.RichHandler + class: kedro.logging.RichHandler + rich_tracebacks: True + # Advance options for customisation. + # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration + # tracebacks_show_locals: False loggers: kedro: diff --git a/tests/framework/project/test_logging.py b/tests/framework/project/test_logging.py index 11cd7f08b3..52e7d5b4c1 100644 --- a/tests/framework/project/test_logging.py +++ b/tests/framework/project/test_logging.py @@ -1,4 +1,4 @@ -# pylint: disable=import-outside-toplevel,unused-import,reimported +# pylint: disable=import-outside-toplevel import logging import sys from pathlib import Path @@ -8,22 +8,28 @@ from kedro.framework.project import LOGGING, configure_logging -default_logging_config = { - "version": 1, - "disable_existing_loggers": False, - "handlers": {"rich": {"class": "rich.logging.RichHandler"}}, - "loggers": {"kedro": {"level": "INFO"}}, - "root": {"handlers": ["rich"]}, -} + +@pytest.fixture +def default_logging_config(): + logging_config = { + "version": 1, + "disable_existing_loggers": False, + "handlers": { + "rich": {"class": "kedro.logging.RichHandler", "rich_tracebacks": True} + }, + "loggers": {"kedro": {"level": "INFO"}}, + "root": {"handlers": ["rich"]}, + } + return logging_config @pytest.fixture(autouse=True) -def reset_logging(): +def reset_logging(default_logging_config): yield configure_logging(default_logging_config) -def test_default_logging_config(): +def test_default_logging_config(default_logging_config): assert LOGGING.data == default_logging_config assert "rich" in {handler.name for handler in logging.getLogger().handlers} assert logging.getLogger("kedro").level == logging.INFO @@ -35,8 +41,9 @@ def test_environment_variable_logging_config(monkeypatch, tmp_path): logging_config = {"version": 1, "loggers": {"kedro": {"level": "WARNING"}}} with config_path.open("w", encoding="utf-8") as f: yaml.dump(logging_config, f) - del sys.modules["kedro.framework.project"] - from kedro.framework.project import LOGGING # noqa + from kedro.framework.project import _ProjectLogging + + LOGGING = _ProjectLogging() assert LOGGING.data == logging_config assert logging.getLogger("kedro").level == logging.WARNING @@ -49,25 +56,91 @@ def test_configure_logging(): assert logging.getLogger("kedro").level == logging.WARNING -def test_rich_traceback_enabled(mocker): - """Note we need to force reload; just doing from kedro.framework.project import ... - will not call rich.traceback.install again. Using importlib.reload does not work - well with other tests here, hence the manual del sys.modules.""" +def test_rich_traceback_enabled(mocker, default_logging_config): rich_traceback_install = mocker.patch("rich.traceback.install") rich_pretty_install = mocker.patch("rich.pretty.install") - del sys.modules["kedro.framework.project"] - from kedro.framework.project import LOGGING # noqa + + LOGGING.configure(default_logging_config) rich_traceback_install.assert_called() rich_pretty_install.assert_called() -def test_rich_traceback_disabled_on_databricks(mocker, monkeypatch): +def test_rich_traceback_not_installed(mocker, default_logging_config): + rich_traceback_install = mocker.patch("rich.traceback.install") + rich_pretty_install = mocker.patch("rich.pretty.install") + rich_handler = { + "class": "kedro.logging.RichHandler", + "rich_tracebacks": False, + } + test_logging_config = default_logging_config + test_logging_config["handlers"]["rich"] = rich_handler + + LOGGING.configure(test_logging_config) + + rich_pretty_install.assert_called_once() + rich_traceback_install.assert_not_called() + + +def test_rich_traceback_configuration(mocker, default_logging_config): + import click + + rich_traceback_install = mocker.patch("rich.traceback.install") + rich_pretty_install = mocker.patch("rich.pretty.install") + + sys_executable_path = str(Path(sys.executable).parent) + traceback_install_defaults = {"suppress": [click, sys_executable_path]} + + rich_handler = { + "class": "kedro.logging.RichHandler", + "rich_tracebacks": True, + "tracebacks_show_locals": True, + } + + test_logging_config = default_logging_config + test_logging_config["handlers"]["rich"] = rich_handler + LOGGING.configure(test_logging_config) + + expected_install_defaults = traceback_install_defaults + expected_install_defaults["show_locals"] = True + rich_traceback_install.assert_called_with(**expected_install_defaults) + rich_pretty_install.assert_called_once() + + +def test_rich_traceback_configuration_extend_suppress(mocker, default_logging_config): + """Test the configuration is not overrided but extend for `suppress`""" + import click + + rich_traceback_install = mocker.patch("rich.traceback.install") + rich_pretty_install = mocker.patch("rich.pretty.install") + + sys_executable_path = str(Path(sys.executable).parent) + traceback_install_defaults = {"suppress": [click, sys_executable_path]} + fake_path = "dummy" + rich_handler = { + "class": "kedro.logging.RichHandler", + "rich_tracebacks": True, + "tracebacks_suppress": [fake_path], + } + + test_logging_config = default_logging_config + test_logging_config["handlers"]["rich"] = rich_handler + LOGGING.configure(test_logging_config) + + expected_install_defaults = traceback_install_defaults + expected_install_defaults["suppress"].extend([fake_path]) + rich_traceback_install.assert_called_with(**expected_install_defaults) + rich_pretty_install.assert_called_once() + + +def test_rich_traceback_disabled_on_databricks( + mocker, monkeypatch, default_logging_config +): monkeypatch.setenv("DATABRICKS_RUNTIME_VERSION", "1") rich_traceback_install = mocker.patch("rich.traceback.install") rich_pretty_install = mocker.patch("rich.pretty.install") - del sys.modules["kedro.framework.project"] - from kedro.framework.project import LOGGING # noqa + + LOGGING.configure(default_logging_config) rich_traceback_install.assert_not_called() rich_pretty_install.assert_called()