Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement DeepEvalEvaluator #346

Merged
merged 10 commits into from
Feb 9, 2024

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented Feb 6, 2024

Related to #250.

We introduce DeepEvalEvaluator, a component that uses the DeepEval LLM evaluation framework to calculate evaluation metrics for RAG pipelines (among others). Refer deepset-ai/haystack#6784 for an overview of the API design.

This PR introduces the following user-facing classes:

  • DeepEvalMetric - A enumeration that lists the supported DeepEval metrics. Currently, only those metrics that are related to RAG pipelines are supported.
  • DeepEvalEvaluator - Th pipeline component interfaces with the evaluation framework. It accepts a single metric and its optional parameters. The inputs to the pipeline are dynamically configured depending on the metric. This is done with help of a metric descriptor table that contains metadata concerning input/output conversion formats, expected inputs/outputs, etc.

The output of the component is a nested list of metric results. Each input can have one or more results, depending on the metric. Each result is a dictionary containing the following keys and values:

  • name - The name of the metric.
  • score - The score of the metric.
  • explanation - An optional explanation of the score.

@shadeMe shadeMe added type:documentation Improvements or additions to documentation new integration Discuss the creation of a new integration in Core integration:deepeval labels Feb 6, 2024
@shadeMe shadeMe requested a review from a team as a code owner February 6, 2024 15:11
@shadeMe shadeMe requested review from masci and removed request for a team February 6, 2024 15:11
@shadeMe shadeMe marked this pull request as draft February 6, 2024 15:21
@shadeMe shadeMe changed the title feat: Implement DeepEvalEvaluator and DeepEvalMetrics feat: Implement DeepEvalEvaluator Feb 7, 2024
Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with deepeval but code looks good to me. I left a question about api docs now that we merged the CI job that will attempt to run hatch run docs on integrations' changes merged on main. Another option might be making that CI job more resilient.

integrations/deepeval/pyproject.toml Show resolved Hide resolved
@shadeMe shadeMe requested a review from masci February 9, 2024 16:22
Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the docs, LGTM

@shadeMe shadeMe merged commit 9bd4417 into deepset-ai:main Feb 9, 2024
10 checks passed
@shadeMe shadeMe deleted the feat/deepeval-evaluator branch February 9, 2024 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration:deepeval new integration Discuss the creation of a new integration in Core topic:CI type:documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants