Add threshold image similarity scores #526
Conversation
harrisonpim
requested review from
alexwlchan and
jamieparkinson
and removed request for
alexwlchan
| val defaultMinScore: Double = similarityMetric match { | ||
| case SimilarityMetric.Blended => 300 | ||
| case SimilarityMetric.Features => 300 | ||
| case SimilarityMetric.Colors => 20 | ||
| } |
There was a problem hiding this comment.
Can you explain how you came up with these default scores? Why is Colors so much lower?
There was a problem hiding this comment.
Sure. There's a notebook in the data science repo which produced the results in wellcomecollection/platform#5581. Based on that analysis, we decided that ~300 seemed like an appropriate threshold for the blended similarity metric.
I re-ran that analysis with the state.inferredData.lshEncodedFeatures and state.inferredData.palette fields individually, and produced these corresponding graphs:
lshEncodedFeatures:
palette:
If my mental maths is right, the scores for colours are generally lower because the state.inferredData.palette field contains fewer, more commonly occurring terms.
There was a problem hiding this comment.
Ah, super. Maybe just include a comment pointing to that ticket, so we can find this again in future?
This PR sets a threshold similarity score for images so that we can avoid displaying questionable matches. Closes #516
This change messes with some of our existing tests - the dummy index populated while setting up the tests can't produce the same scores as the fully populated prod index, and the thresholds therefore filter out any results which might have been matched. In some tests I've been able to manually set the
minScoreto 0, but in others which call the API itself, that's not possible. I feel like these things might be more appropriately tested inrank, where tests can be run against a full index and scoring can be properly examined.