Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Integration of 3rd party evaluation frameworks #6784

Merged
merged 2 commits into from
Jan 22, 2024

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented Jan 19, 2024

Proposed Changes:

Proposal for implementing third-party LLM evaluation frameworks in haystack-core-integrations.

Rendered.

Checklist

@shadeMe shadeMe added proposal 2.x Related to Haystack v2.0 labels Jan 19, 2024
@shadeMe shadeMe requested review from a team as code owners January 19, 2024 13:05
@shadeMe shadeMe requested review from dfokina and julian-risch and removed request for a team January 19, 2024 13:05
@shadeMe shadeMe force-pushed the proposal/model-based-eval-components branch from 32dcbd9 to b58ccc9 Compare January 19, 2024 13:06
@shadeMe shadeMe added the proposal:review Proposal is in "Review" state label Jan 19, 2024
@sandangel
Copy link

sandangel commented Jan 19, 2024

Is it possible to also add https://github.com/Arize-ai/phoenix (both evaluation and tracing) ? I think they have plan to integrate with haystack for tracing but postponed it to wait for haystack 2.0

@julian-risch
Copy link
Member

@sandangel Yes, we are not limiting the integrations to the three frameworks mentioned in this proposal. We'll start with one framework as an example and adding more is encouraged through similar integrations. I can imagine a PhoenixEvaluator component as a Haystack integration based on https://github.com/Arize-ai/phoenix?tab=readme-ov-file#llm-evals 👍

@sandangel
Copy link

That sounds great. Thanks for putting this together.

@shadeMe shadeMe merged commit 5c8feea into deepset-ai:main Jan 22, 2024
6 checks passed
@shadeMe shadeMe deleted the proposal/model-based-eval-components branch January 22, 2024 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 proposal:review Proposal is in "Review" state proposal
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants