5740 metrics reloaded support

[brudfors]

Signed-off-by: Mikael Brudfors [email protected]

part of #5740

Description

Makes the binary and categorical metrics from MetricsReloaded available in MONAI via a wrapper module (monai/metrics/wrapper.py).

This module allows to use the MetricsReloaded metrics as, e.g.:

import torch
from monai.metrics import MetricsReloadedBinary

metric_name = "Cohens Kappa"
metric = MetricsReloadedBinary(metric_name=metric_name)

# first iteration
# shape [batch=1, channel=1, 2, 2]
y_pred = torch.tensor([[[[1.0, 0.0], [0.0, 1.0]]]])
y = torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]])
print(metric(y_pred, y))

# second iteration
# shape [batch=1, channel=1, 2, 2]
y_pred = torch.tensor([[[[1.0, 0.0], [0.0, 0.0]]]])
y = torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]])
print(metric(y_pred, y))

# aggregate
# shape ([batch=2, channel=1])
print(metric.aggregate(reduction="none"))  # tensor([[0.5], [0.2]])

# reset
metric.reset()

Tests of all metrics are in tests/test_metrics_reloaded.py.

Note that MetricsReloaded is an optional dependency of MONAI; so in order to use the wrapper, MONAI needs to be installed with:

pip install '.[metricsreloaded]'

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[wyli]

/black
it looks great, any feedback and things we can improve for better integration (from both monai and MetricsReloaded perspectives)? I think once we confirm MetricsReloaded's API is stable, this PR can be merged

[brudfors]

it looks great, any feedback and things we can improve for better integration (from both monai and MetricsReloaded perspectives)? I think once we confirm MetricsReloaded's API is stable, this PR can be merged

Sounds good!

This PR is limited to supporting the semantic segmentation metrics from MetricsReloaded and for that I cannot, for now, think of any improvements on the MONAI side. This might change when we integrate the instance segmentation and object detection metrics, later down the line. On the MetricsReloaded side, there are still some improvements that can be made, but I don't think they would require changing the API that MONAI uses.

[wyli]

/build

[wyli]

We'll add the ignite wrapper in a separate pr

[wyli]

/build