[Bug]: Unable to set Safety settings to model using Vertex GenAI evals

[MarkEdmondson1234]

File Name

https://cloud.google.com/vertex-ai/generative-ai/docs/models/run-evaluation

What happened?

When I bring my own data, I can't specify a model otherwise I get an error. However, if I don't specify my own model, it uses a default that has safety settings set to None, which I do not qualify for. eg.

...
  eval_dataset = pd.DataFrame({
        "response" : [trace.output],
        "prompt" : [trace.input]
    })

    eval_task = EvalTask(
        dataset=eval_dataset,
        metrics=metrics,
        experiment="multivac-evals"
    )

    eval_result: EvalResult = eval_task.evaluate()

The error is due to the GenerationModel used with safety settings BLOCK_NONE, that is done within my project but I can not control.

When I try to specify a model (with BLOCK_HIGH safety settings) in evaluate(model=model) it won't permit it.

I would like either:

1. to be able to use my own model object (perhaps eval_model?)
2. or have the default model use BLOCK_HIGH safety setting defaults.

Relevant log output

When running with no `model` specified:

    Error: /PredictionServiceV1.GenerateContent to [2002:a05:7512:803::]:4084 : APP_ERROR(3) User has requested a restricted HarmBlockThreshold setting BLOCK_NONE. You can get access either (a) through an allowlist via your Google account team, or (b) by switching your account type to monthly invoiced billing via this instruction: https://cloud.google.com/billing/docs/how-to/invoiced-billing..

When running with own model (which has BLOCK_HIGH)

    The `model` parameter or `baseline_model` in pairwise metric is specified, but the evaluation `dataset` contains model response column or baseline model response column `response` to perform bring-your-own-response(BYOR) evaluation. If you would like to perform evaluation using the dataset with the existing model response column or or baseline model response column `response`, please remove `model` parameter in `EvalTask.evaluate()` function or `baseline_model` in `PairwiseMetric`

Code of Conduct

• I agree to follow this project's Code of Conduct

[jsondai]

When running with no model specified: "User has requested a restricted HarmBlockThreshold setting BLOCK_NONE. You can get access either (a) through an allowlist via your Google account team, or (b) by switching your account type to monthly invoiced billing via this instruction: https://cloud.google.com/billing/docs/how-to/invoiced-billing."

1. Thanks for reporting this. A fix is created and deployed. This should be resolved now. Could you try this again?

When running with own model (which has BLOCK_HIGH): The model parameter or baseline_model in pairwise metric is specified, but the evaluation dataset contains model response column or baseline model response column response to perform bring-your-own-response(BYOR) evaluation. If you would like to perform evaluation using the dataset with the existing model response column or or baseline model response column response, please remove model parameter in EvalTask.evaluate() function or baseline_model in PairwiseMetric

2. The issue is that there is a column named "response" in your dataset. If you remove this column, then the GenAI Eval SDK can use the input model (BLOCK_HIGH is not required.) to create a new response for each prompt input of your dataset for evaluation.

[MarkEdmondson1234]

@jsondai thanks for quick turnaround! Will try it out. Regarding 2. I want to check my pre-made responses, so was only adding my own model parameter not to test that model, but to work around the default model being blocked assessing my pre-done responses.

[MarkEdmondson1234]

@jsondai its working now, thanks :)