Skip to content

Commit

Permalink
[Security Assistant] Add support for manually entering eval datasets (#…
Browse files Browse the repository at this point in the history
…194072)

## Summary

On cloud environments we don't send the LangSmith credentials through
the `GET /evaluate` route which returns available datasets for
selection, so the datasets are never populated. Since the Dataset field
didn't allow custom options, this means you couldn't perform evals in
cloud environments.

This PR updates the Dataset field to take custom options so that you can
manually enter the dataset name in cloud environments:

<p align="center">
<img width="500"
src="https://github.com/user-attachments/assets/4828e085-180c-42bb-9656-34bda57b74b5"
/>
</p> 


To test, enable the below feature flag to show the evaluation tab under
settings:

```
xpack.securitySolution.enableExperimental:
  - "assistantModelEvaluation"
```
  • Loading branch information
spong authored Sep 26, 2024
1 parent ce08d4e commit 1c0ec85
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,20 @@ export const EvaluationSettings: React.FC = React.memo(() => {
},
[setSelectedDatasetOptions]
);
const onDatasetCreateOption = useCallback(
(searchValue: string) => {
const normalizedSearchValue = searchValue.trim().toLowerCase();
if (!normalizedSearchValue) {
return;
}
const newOption = {
label: searchValue,
};

setSelectedDatasetOptions([newOption]);
},
[setSelectedDatasetOptions]
);

// Predictions
// Connectors / Models
Expand Down Expand Up @@ -244,6 +258,7 @@ export const EvaluationSettings: React.FC = React.memo(() => {
options={datasetOptions}
selectedOptions={selectedDatasetOptions}
onChange={onDatasetOptionsChange}
onCreateOption={onDatasetCreateOption}
compressed={true}
/>
</EuiFormRow>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,8 @@ export const EVALUATOR_DATASET_LABEL = i18n.translate(
export const LANGSMITH_DATASET_DESCRIPTION = i18n.translate(
'xpack.elasticAssistant.assistant.settings.evaluationSettings.langsmithDatasetDescription',
{
defaultMessage: 'Name of dataset hosted on LangSmith to evaluate.',
defaultMessage:
'Name of dataset hosted on LangSmith to evaluate. Must manually enter on cloud environments.',
}
);

Expand Down

0 comments on commit 1c0ec85

Please sign in to comment.