You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The validate_metric_parameters function doesn't actually check if an empty dict is passed to it because of the set subset function (which always returns true for empty sets). We should make this function similar to DeepEval's.
Also we should not convert Nones passed as the metric_param to an empty dict - The none should handled explicitly.
Our unit tests don't catch this atm - needs to be fixed.
Furthermore, i see that the unit test relies on the Ragas backend raising an error (for aspect critique, for example) - We shouldn't do this, especially when upstream breaks so often.
Let's modify the impl so that our scaffolding checks the init parameters and our unit tests reliably catch those.
We'll also need to test the other metrics that require init params.
Since those are currently missing, we missed the above issue with the validation logic.
The init parameters in the descriptors seem to be incorrect. AFAICT, these are the only metrics that need extra init params:
Answer correctness - weights
Answer similarity - threshold
Aspect critique - name, definition, strictness
Answer relevancy - strictness
The rest need to be removed. Keep in mind that the docstrings in Ragas seem to be out of sync.
The text was updated successfully, but these errors were encountered:
The text was updated successfully, but these errors were encountered: