Fix Classification Performance

[czaloom]

Changes

• Fixed a bug created in PR Add support for new filtering ops #581 where the default metric computation for classification was slowed down.
• Refactored the PRCurve and DetailedPRCurve computation for classification.
  • PRCurve and DetailedPRCurve take the same time to compute and could be merged if desired.
  • Computation time is of the same magnitude as the time to compute default classification metrics

Performance

Before (v0.27.3)

{'limit': 10, 'ingest': '0.2871', 'base': '1.0894', 'base+pr': '1.0716', 'base+pr+detailed pr': '1.0793'}
{'limit': 10, 'ingest': '0.3532', 'base': '1.0686', 'base+pr': '1.0680', 'base+pr+detailed pr': '1.0741'}
{'limit': 100, 'ingest': '0.5643', 'base': '1.0755', 'base+pr': '2.0900', 'base+pr+detailed pr': '2.0856'}
{'limit': 100, 'ingest': '0.5666', 'base': '1.0735', 'base+pr': '2.0858', 'base+pr+detailed pr': '2.0892'}
{'limit': 1000, 'ingest': '4.7223', 'base': '32.5760', 'base+pr': '61.8423', 'base+pr+detailed pr': '62.1851'}
{'limit': 1000, 'ingest': '4.7074', 'base': '32.5851', 'base+pr': '60.7616', 'base+pr+detailed pr': '60.5008'}
{'limit': 5000, 'ingest': '23.0799', 'base': '762.0845', 'base+pr': '4081.4385', 'base+pr+detailed pr': '4062.1859'}

After

{'limit': 10, 'ingest': '0.2971', 'base': '1.0675', 'base+pr': '1.0577', 'base+pr+detailed pr': '1.0640'}
{'limit': 10, 'ingest': '0.2607', 'base': '1.0516', 'base+pr': '1.0505', 'base+pr+detailed pr': '1.0597'}
{'limit': 100, 'ingest': '0.5761', 'base': '1.0643', 'base+pr': '1.0632', 'base+pr+detailed pr': '1.0648'}
{'limit': 100, 'ingest': '0.5290', 'base': '1.0557', 'base+pr': '1.0657', 'base+pr+detailed pr': '1.0723'}
{'limit': 1000, 'ingest': '4.9709', 'base': '3.1396', 'base+pr': '4.2505', 'base+pr+detailed pr': '4.3072'}
{'limit': 1000, 'ingest': '4.9524', 'base': '3.1360', 'base+pr': '4.2255', 'base+pr+detailed pr': '5.4064'}
{'limit': 5000, 'ingest': '29.3421', 'base': '9.2979', 'base+pr': '15.8753', 'base+pr+detailed pr': '15.7665'}
{'limit': 5000, 'ingest': '28.7270', 'base': '11.3355', 'base+pr': '17.9243', 'base+pr+detailed pr': '18.0771'}

# showing consistency
{'limit': 5000, 'ingest': '28.2676', 'base': '7.3984', 'base+pr': '13.7927', 'base+pr+detailed pr': '13.8671'}
{'limit': 5000, 'ingest': '28.2710', 'base': '7.2600', 'base+pr': '13.7537', 'base+pr+detailed pr': '14.0668'}
{'limit': 5000, 'ingest': '27.7182', 'base': '7.2585', 'base+pr': '13.8196', 'base+pr+detailed pr': '13.8971'}
{'limit': 5000, 'ingest': '27.6027', 'base': '7.2584', 'base+pr': '13.6964', 'base+pr+detailed pr': '14.0638'}
{'limit': 5000, 'ingest': '27.8308', 'base': '9.4128', 'base+pr': '15.7377', 'base+pr+detailed pr': '15.7828'}
{'limit': 5000, 'ingest': '27.9315', 'base': '10.2762', 'base+pr': '16.9073', 'base+pr+detailed pr': '16.7292'}
{'limit': 5000, 'ingest': '27.8125', 'base': '11.3019', 'base+pr': '17.8417', 'base+pr+detailed pr': '17.8011'}
{'limit': 5000, 'ingest': '24.3069', 'base': '13.4752', 'base+pr': '19.8883', 'base+pr+detailed pr': '19.9296'}
{'limit': 5000, 'ingest': '27.9290', 'base': '15.4618', 'base+pr': '21.7758', 'base+pr+detailed pr': '21.8493'}
{'limit': 5000, 'ingest': '27.9097', 'base': '16.3366', 'base+pr': '22.9540', 'base+pr+detailed pr': '24.0885'}
{'limit': 5000, 'ingest': '27.8893', 'base': '18.4587', 'base+pr': '23.9570', 'base+pr+detailed pr': '24.8777'}

[ntlind]

A prediction having a score less than the threshold is still a valid prediction though

what is the point of the score threshold in that case?

the score threshold is meant to mean "only consider predictions with a score greater than x to be valid predictions"

[czaloom]

The point of a score threshold is to determine whether the prediction is positive vs negative.

Whether that prediction is correct determines its truth (True, False).

Combine these and you get TP, FP, FN and TN.

The variation of missing_detection doesnt really map well to the classification task (as compared to the obj det task) as we enforce the existence of predictions to groundtruths at ingestion time. (See validate_matching_label_keys)

This logic also applies to hallucination for FP, which, if you look at that test never gets a value counted.

[czaloom]

I reached out to Matt and I think this is a definition issue. Missing detection doesnt make sense for classification. The condition of FN that you are referring to fits something closer to a "no winner" condition.

Matt suggested "No prediction" and im wondering if "Null Prediction" would make more sense.

How does all this sound to you?

[ntlind]

A prediction having a score less than the threshold is still a valid prediction though

what is the point of the score threshold in that case?

the score threshold is meant to mean "only consider predictions with a score greater than x to be valid predictions"