Fix OD benchmarks and disallow calculating DetailedPRCurves when using AnnotationType.RASTER #726
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improvements
valor_core
to calculateDetailedPrecisionRecallCurve
NotImplementedError
when a user tries to calculateDetailedPrecisionRecallCurve
usingAnnotationType.RASTER
.valor_core
will try to produce (datum_uid, mask) pairs, which easily leads to OOM issues since we're creating sets of tuples, each containing relatively large masksNotImplementedError
for now. when we re-prioritize DetailedPRCurves in the future, we can discuss what the right output is for this metric.compute_detailed
for the Box and Polygon OD benchmarks since a) they make the tests take a while to run (~80 seconds for just one run of 5000Box
datums; 213 seconds for one Polygon run) and b) profiling and improving these PR curves is low priority right now. We can try adding them back after Justin merges his PRs.valor_core
OD benchmarks to use the ValorDetectionManager class