Bump torchmetrics from 0.5.1 to 0.11.0

[dependabot]

Bumps torchmetrics from 0.5.1 to 0.11.0.

Release notes

Sourced from torchmetrics's releases.

Adding Multimodal and nominal domain

We are happy to announce that Torchmetrics v0.11 is now publicly available. In Torchmetrics v0.11 we have primarily focused on the cleanup of the large classification refactor from v0.10 and adding new metrics. With v0.11 are crossing 90+ metrics in Torchmetrics nearing the milestone of having 100+ metrics.

New domains

In Torchmetrics we are not only looking to expand with new metrics in already established metric domains such as classification or regression, but also new domains. We are therefore happy to report that v0.11 includes two new domains: Multimodal and nominal.

Multimodal

If there is one topic within machine learning that is hot right now then it is generative models and in particular image-to-text generative models. Just recently stable diffusion v2 was released, able to create even more photorealistic images from a single text prompt than ever

In Torchmetrics v0.11 we are adding a new domain called multimodal to support the evaluation of such models. For now, we are starting out with a single metric, the CLIPScore from this paper that can be used to evaluate such image-to-text models. CLIPScore currently achieves the highest correlation with human judgment, and thus a high CLIPScore for an image-text pair means that it is highly plausible that an image caption and an image are related to each other.

Nominal

If you have ever taken any course in statistics or introduction to machine learning you should hopefully have heard about data can be of different types of attributes: nominal, ordinal, interval, and ratio. This essentially refers to how data can be compared. For example, nominal data cannot be ordered and cannot be measured. An example, would it be data that describes the color of your car: blue, red, or green? It does not make sense to compare the different values. Ordinal data can be compared but does have not a relative meaning. An example, would it be the safety rating of a car: 1,2,3? We can say that 3 is better than 1 but the actual numerical value does not mean anything.

In v0.11 of TorchMetrics, we are adding support for classic metrics on nominal data. In fact, 4 new metrics have already been added to this domain:

• CramersV
• PearsonsContingencyCoefficient
• TschuprowsT
• TheilsU

All metrics are measures of association between two nominal variables, giving a value between 0 and 1, with 1 meaning that there is a perfect association between the variables.

Small improvements

In addition to metrics within the two new domains v0.11 of Torchmetrics contains other smaller changes and fixes:

• TotalVariation metric has been added to the image package, which measures the complexity of an image with respect to its spatial variation.

• MulticlassExactMatch metric has been added to the classification package, which for example can be used to measure sentence level accuracy where all tokens need to match for a sentence to be counted as correct

• KendallRankCorrCoef have been added to the regression package for measuring the overall correlation between two variables

• LogCoshError have been added to the regression package for measuring the residual error between two variables. It is similar to the mean squared error close to 0 but similar to the mean absolute error away from 0.

---

Finally, Torchmetrics now only supports v1.8 and higher of Pytorch. It was necessary to increase from v1.3 to secure because we were running into compatibility issues with an older version of Pytorch. We strive to support as many versions of Pytorch, but for the best experience, we always recommend keeping Pytorch and Torchmetrics up to date.

---

[0.11.0] - 2022-11-30

Added

• Added MulticlassExactMatch to classification metrics (#1343)
• Added TotalVariation to image package (#978)
• Added CLIPScore to new multimodal package (#1314)
• Added regression metrics:

... (truncated)

Changelog

Sourced from torchmetrics's changelog.

[0.11.0] - 2022-11-30

Added

• Added MulticlassExactMatch to classification metrics (#1343)
• Added TotalVariation to image package (#978)
• Added CLIPScore to new multimodal package (#1314)
• Added regression metrics:
  • KendallRankCorrCoef (#1271)
  • LogCoshError (#1316)
• Added new nominal metrics:
  • CramersV (#1298)
  • PearsonsContingencyCoefficient (#1334)
  • TschuprowsT (#1334)
  • TheilsU (#1337)
• Added option to pass distributed_available_fn to metrics to allow checks for custom communication backend for making dist_sync_fn actually useful (#1301)
• Added normalize argument to Inception, FID, KID metrics (#1246)

Changed

• Changed minimum Pytorch version to be 1.8 (#1263)
• Changed interface for all functional and modular classification metrics after refactor (#1252)

Removed

• Removed deprecated BinnedAveragePrecision, BinnedPrecisionRecallCurve, RecallAtFixedPrecision (#1251)
• Removed deprecated LabelRankingAveragePrecision, LabelRankingLoss and CoverageError (#1251)
• Removed deprecated KLDivergence and AUC (#1251)

Fixed

• Fixed precision bug in pairwise_euclidean_distance (#1352)

[0.10.3] - 2022-11-16

Fixed

• Fixed bug in Metrictracker.best_metric when return_step=False (#1306)
• Fixed bug to prevent users from going into an infinite loop if trying to iterate of a single metric (#1320)

[0.10.2] - 2022-10-31

Changed

• Changed in-place operation to out-of-place operation in pairwise_cosine_similarity (#1288)

Fixed

• Fixed high memory usage for certain classification metrics when average='micro' (#1286)
• Fixed precision problems when structural_similarity_index_measure was used with autocast (#1291)

... (truncated)

Commits

• bc057d4 releasing 0.11.0
• ed2249d improved retrieval doc (#1358)
• 6dc67df Fix wrong member naming in cer/wer (#1357)
• cf013b5 CI: fix using actual TM version integrations (#1353)
• 20eab43 Code cleaning after classification refactor 2/n (#1252)
• d66cbc3 Fix precision issue in pairwise euclidean distance (#1352)
• 339a7a6 mergify: drop ready for draft/wip (#1350)
• 68563ad Cleanup of deprecated code (#1349)
• 7d77daf ci: reruns failed tests (#1348)
• 68a6990 Exact match multiclass (#1343)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[borchero]

@dependabot rebase

[marcovarrone]

Tested and seems to work fine.
The reason why it's better not to use the caret symbol "^" for torchmetrics is that for packages that are still at version 0.x, the caret allows for all the versions of 0.x.y but not 0.(x+1).
This pull request allows using more recent versions of pytorch-lightning such as version 1.8.0.

[borchero]

Ah nice, I didn't know the caret behaves differently for 0.x but makes a lot of sense! Thanks for explaining @marcovarrone 👻