feat(gen ai): showcase different options for computation-based metric #12756
Conversation
product-auto-label
bot
added
the
samples
| @@ -37,7 +39,37 @@ def get_rouge_score() -> EvalResult: | |||
| life, including endangered species, it faces serious threats from | |||
| climate change, ocean acidification, and coral bleaching.""" | |||
|
|
|||
| # Compare pre-generated model responses against the reference (ground truth). | |||
| # Option1: Run model inference and evaluate model response against the reference (ground truth) | |||
There was a problem hiding this comment.
The code samples looks too big now!
There was a problem hiding this comment.
Yep, I understand
| @@ -37,7 +39,37 @@ def get_rouge_score() -> EvalResult: | |||
| life, including endangered species, it faces serious threats from | |||
| climate change, ocean acidification, and coral bleaching.""" | |||
|
|
|||
| # Compare pre-generated model responses against the reference (ground truth). | |||
| # Option1: Run model inference and evaluate model response against the reference (ground truth) | |||
There was a problem hiding this comment.
@msampathkumar , I'm thinking about showcasing 2 different options of using the computation-based metrics — Bring-your-own-response (BYOR) and with running model inference.
The reason is that for me, as a developer, the line between these options wasn't immediately obvious (hence this issue with the "prompt" column being silently unused), so I want to make it crystal-clear.
There was a problem hiding this comment.
While I understand your point, this code samples is still too big(100 lines). Let me check with the tech writing team.
There was a problem hiding this comment.
Also note, I don't see any example response section for this part of the code.
Description
Fixes #
Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.
Checklist
nox -s py-3.9(see Test Environment Setup)nox -s lint(see Test Environment Setup)