Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RAGAS] Update metric parameter validation and metric descriptors #528

Closed
julian-risch opened this issue Mar 4, 2024 · 0 comments · Fixed by #555
Closed

[RAGAS] Update metric parameter validation and metric descriptors #528

julian-risch opened this issue Mar 4, 2024 · 0 comments · Fixed by #555

Comments

@julian-risch
Copy link
Member

  • The validate_metric_parameters function doesn't actually check if an empty dict is passed to it because of the set subset function (which always returns true for empty sets). We should make this function similar to DeepEval's.
    • Also we should not convert Nones passed as the metric_param to an empty dict - The none should handled explicitly.
    • Our unit tests don't catch this atm - needs to be fixed.
    • Furthermore, i see that the unit test relies on the Ragas backend raising an error (for aspect critique, for example) - We shouldn't do this, especially when upstream breaks so often.
      • Let's modify the impl so that our scaffolding checks the init parameters and our unit tests reliably catch those.
      • We'll also need to test the other metrics that require init params.
        • Since those are currently missing, we missed the above issue with the validation logic.
  • The init parameters in the descriptors seem to be incorrect. AFAICT, these are the only metrics that need extra init params:
    • Answer correctness - weights
    • Answer similarity - threshold
    • Aspect critique - name, definition, strictness
    • Answer relevancy - strictness
  • The rest need to be removed. Keep in mind that the docstrings in Ragas seem to be out of sync.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants