diff --git a/docs/plugins/vertex-ai.md b/docs/plugins/vertex-ai.md index b5b7de818..464eb4302 100644 --- a/docs/plugins/vertex-ai.md +++ b/docs/plugins/vertex-ai.md @@ -7,6 +7,13 @@ through the [Vertex AI API](https://cloud.google.com/vertex-ai/generative-ai/doc - Imagen2 image generation - Gecko text embedding generation +It also provides access to subset of evaluation metrics through the Vertex AI [Rapid Evaluation API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation). + +- [Safety](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#safetyinput) +- [Groundeness](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#groundednessinput) +- [ROUGE](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#rougeinput) +- [BLEU](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#bleuinput) + ## Installation ```posix-terminal @@ -66,6 +73,8 @@ credentials. ## Usage +### Generative AI Models + This plugin statically exports references to its supported generative AI models: ```js @@ -115,7 +124,7 @@ const embedding = await embed({ }); ``` -### Anthropic Claude 3 on Vertex AI Model Garden +#### Anthropic Claude 3 on Vertex AI Model Garden If you have access to Claude 3 models ([haiku](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-haiku), [sonnet](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-sonnet) or [opus](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-opus)) in Vertex AI Model Garden you can use them with Genkit. @@ -147,3 +156,36 @@ const llmResponse = await generate({ prompt: 'What should I do when I visit Melbourne?', }); ``` + +### Evaluators + +To use the evaluators from Vertex AI Rapid Evaluation, add an `evaluation` block to your `vertexAI` plugin configuration. + +```js +import { vertexAI, VertexAIEvaluationMetricType } from '@genkit-ai/vertexai'; + +export default configureGenkit({ + plugins: [ + vertexAI({ + projectId: 'your-cloud-project', + location: 'us-central1', + evaluation: { + metrics: [ + VertexAIEvaluationMetricType.SAFETY, + { + type: VertexAIEvaluationMetricType.ROUGE, + metricSpec: { + rougeType: 'rougeLsum', + }, + }, + ], + }, + }), + ], + // ... +}); +``` + +The configuration above adds evaluators for the `Safety` and `ROUGE` metrics. The example shows two approaches- the `Safety` metric uses the default specification, whereas the `ROUGE` metric provides a customized specification that sets the rouge type to `rougeLsum`. + +Both evaluators can be run using the `genkit eval:run` command with a compatible dataset: that is, a dataset with `output` and `reference` fields. The `Safety` evaluator can also be run using the `genkit eval:flow -e vertexai/safety` command since it only requires an `output`. diff --git a/js/plugins/vertexai/src/evaluation.ts b/js/plugins/vertexai/src/evaluation.ts index c8fa1937e..f5a9195e8 100644 --- a/js/plugins/vertexai/src/evaluation.ts +++ b/js/plugins/vertexai/src/evaluation.ts @@ -25,6 +25,7 @@ import { EvaluatorFactory } from './evaluator_factory'; * https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation#parameter-list */ export enum VertexAIEvaluationMetricType { + // Update genkit/docs/plugins/vertex-ai.md when modifying the list of enums SAFETY = 'SAFETY', GROUNDEDNESS = 'GROUNDEDNESS', BLEU = 'BLEU',