Added LLM evaluation.

Co-authored-by: kukushking <[email protected]>
Co-authored-by: ViktorMalesevic <[email protected]>
Co-authored-by: Marco1402 <[email protected]>
Co-authored-by: omsh <[email protected]>

CHANGELOG.md

@@ -24,9 +24,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - fix inputs for `bedrock-finetuning` module not working
 - add `retention-type` argument for the bucket in the `bedrock-finetuning` module
 - fix broken dependencies for `examples/airflow-dags`
-- use `add_dependency` to avoid deprecation warnings from CDK.
-- Various typo fixes.
-- Various clean-ups to the SageMaker Service Catalog templates.
+- use `add_dependency` to avoid deprecation warnings from CDK
+- various typo fixes
+- various clean-ups to the SageMaker Service Catalog templates
+- fix opensearch removal policy
 
 ## v1.2.0
 

manifests/fmops-qna-rag/storage-modules.yaml

@@ -4,7 +4,7 @@ parameters:
   - name: encryption-type
     value: SSE
   - name: retention-type
-    value: RETAIN
+    value: DESTROY
   - name: vpc-id
     valueFrom:
       moduleMetadata:

modules/sagemaker/sagemaker-templates-service-catalog/README.md

@@ -18,6 +18,12 @@ The template contains an example SageMaker Pipeline to train a model on Abalone
 
 ![Abalone with XGBoost](docs/_static/abalone-xgboost-template.png "Abalone with XGBoost Template Architecture")
 
+#### LLM evaluate template
+
+This project template contains SageMaker pipeline that performs LLM evaluation. 
+
+![LLM evaluate template](docs/_static/llm-evaluate.png "LLM Evaluate Template Architecture")
+
 The template is based on basic multi-account template from [AWS Enterprise MLOps Framework](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/README.md#sagemaker-project-stack).
 
 #### Batch Inference Template

modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/llm-evaluate.xml

@@ -0,0 +1 @@
+<mxfile modified="2024-04-15T13:30:57.955Z" host="design-inspector.a2z.com" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" etag="LSdO9LAzmekkD11aoc1V" version="10.1.8" type="device"><diagram id="8qc0D0XUhG9jV8B8jlGXA" name="Page-1">7V1rd5s4E/41+agcQOL20Y7bbk7bPXmb7WX3yx4BwqbBxgtyYvfXvyMDNtgYOw22wVaak1oD6K6ZZx6NxQ2+G88/xHQ6+hx5LLzRFG9+gwc3mmYbBP4KwSIVGHomGMaBl4rUteAx+MUyoZJJZ4HHktKNPIpCHkzLQjeaTJjLSzIax9FL+TY/CsulTumQbQkeXRpuS78HHh+lUksz1/I/WDAc5SWrhp1eGdP85qwlyYh60UtBhN/d4Ls4inj6aTy/Y6Hou7xfsucmbM7FlXvvGw1nWbWIscrhC0uiWeyyAUvcOJjyKIaH4kyY3v3vDe7ltWBxQMPgF+VBNEHPLE7g//Su5+wWmnVZXJFxVuYjG9MJD9wB5fQumnAaTFh8SO7p0zwOJsNPAWcxDdOx42zCS62extGUxTybNSPOxXj3brT38Au3R2E0XNwmzJ3FAV/c0jH9FU1uPfYMl/1oNvGWNYCEF9BhTMfoOUhmq5qBnGqartu6gVzs+Yi4moscYtmQVEzfsKjj2ukQv0/rfP/lfqtbX1UrmMXBcIKCSTKFeSr68r0bjafRBFqeQMIi1FIc3Ue6QTREqGoiW7d0xBzfcwzdJw5zm+2aZJFwNkZjsWRhPECiEB3bum0iolkYEd/QkaXYCqLMMH3bw8wmdrFT4EP1TMivVszN/FI23aun/mrVxeV58ZoJr3Zkwqtywl/dhI+cn8JYaUpIHbCXy3Zl0xUmhhuEfy2mWZ8PGeQfuGjVd+l9+Ur5qH0f9T/+p3z7rkW//oEBpuM//0DqauqtllbCF7lNi0WnMJGBcoP7LyOYl49TuhzkFzDiIBvxsaiVCh/9IAzvoIuhIYMJVABEHk1Gy8fF9bQ6MPM5m+9cvJko9vyvU5idaeEFQ50kjIsmfysuIFxoxAcWjRmPF3BzXoqerd5sWphZ8mVtpU0jk40KFhrrG6t+uMp6Pc4Hqpiazu+A4rH+c5WB8uPr6IP985vVt36GH370TqOMFN11day6yNOIiojtYlh3HmgkD5um6epEM3ypjPYoo6N3CvF11bZVH/nYNaBTCHSKoyhIUbFuKNSxNZ/s6pRVHV5eXm5f8G0UiyZCdrZoqWiEpiHQByhZwKSeowkUiLnQek22sNTjQ+GgoITHM5fPYibq4RDGqGshhTAdEQYDbhm2mIa64Ts2MR3TOWoDa9dnrXZpG1xoyjKK6qfGcaehfASf6zN9WqrBh2Aqlg94iNXW83nqHmIv8dqdqjKY0ygQKwI+6334Bf16p9zocOVOpG41fUOwmTbLAnU7JfIoCzbTZlmgbmavbpSvblawINhKlbJXNspXChWEX9yPZlz0+d3K5RYwQpj1AObuBlYoIIkqoOHDPMmcflXL01nHi1zBaV4O5Xi+XL239CXRlNshIJjpssx7d2n1Ki7/K4YeMuBx9MTySsES1IhlqaQa2AgUE4Dv3wtBL4KYR6IUmqVC5nORI7RALNplaoCVrNaFInq9vtm3ikBJOSNQIqpeAkoa3kZKxNC3kZJqHBsppctOQiUJlSRUklDpDVCpBhIcWh1NURRRHdC075d1cUegT6AiKfJ4G2jLFd0Vo7Z99MaIApoIDwJrtViNeUOWI4go5qNoGE1o+G4t7ZfpjwJA+ck4X2RohM54JKDOKodPkQADBcpDFLTPjhdXI9Wgj9Wn/6GHiERfEyX5db9o0tzHLIRp8Mw2ctg23dmjDwLUrmGCppT5lC3rz2k8ZDx7qjSFCtVoABN0BBLU1r5t61yCggpQIBHfYabf0FXdNDQFmcSBzjUVBlWxPeTYjq8bjuE65Limvx/SydOfMND3g0pFUqNZmxqr7V7xVN9VDIMhbBlixjMFWVghyHCZTwDtKTQjmltjaqmbqca1gd3ZczvNa+6Qz8Zhb5nf2mv+JGrwECUBT5WpE3EejeGGZdX61H0aLk1vwVP2lz87Pe+Cba7kHM7lUpsbprLCo8YVWw8N+tM1U771lvNqVeu2yaKWqyue5yFKbQsRy2WIqtRFOrV1H1xKwsyGEUULfOlOe5onsDVX7dxp+rG8OzYP+I/C57/FLbd6lhrMsyeWiUUh8QBNADUt+i2VTUBl/ygmCjmJ5DqrZWpRTG1mdlyXc8cOUha7ljp0tYNxFv+0EWpZ0ztiDeur3zadIh1J6UheriNZry/b40v+XjuajjSsGGDVdBhYKeT6sG4JVVRETd1Dlso8TImp6Vg7G0IxQrF76wXQDmPIlzY1FTm54A569C4ajwPx7Bc2Fa5sJGxSeh+U4mw+u1x9hRwrYZAL+bpZvvsRUDrdXheLUBmPUBWTUBmXsB2bULptGS1QUcKmsEpmbgvV7dvyAINtYZWsKppi82m14ml14+ndsQwb+/nw770Yj60YB7hGBiZcLFwbBDFklMKAicByG6EG8Az8kL5dFZyw4kNKkQN7CZb6uAUXaiXWVCmqc0/cBV2aZ0j4wVzUY0cgRq560jCMPiQrAzIKC+B8pM3WBoe2zdpYFayNdewoiNy+dBSqNm0eJVRtM3kjAyFaTU/thDBnCIXokOfTTaqMHIsqOwsrZV0DK0U6beq1napbmvrWmfpL0c2SlarVl5KVunBWKo9nhSd74Ob71BUqyaBj4eNPnGRapqP2UlT7S3wEW6n0Z+4T2OG3cl8JPojzsiTn1R3Oy1B6GJuv47w001RV44o4rwS3iOvSiN4SrsvqNABu2uh2BgAzRoiDbYpc4jiIKABrHNtQEcAAi7mqjVk+ddoDgK+bCaqCDSXDLhmhi2OE8NGCp87BCOX7MwcwQmZnGSHc7TglLOOUJCMkGaFWMEJNH513Nkbo6PFWHWeExCZffxaEXiORSU6a0wH7TWtLIUma1pM0MjDpwMCkdP6f8XyWjYPsWhOXtDq4tpvQVJ4k2yFoKuOSLpuNKkAWyTtdHO+k1R+f95Yv7XX9y3arQ+v3k1hKZ0ksrSvnuNVXXyKFDiCFS1H0ksQ6UzjQiUiso5NxF0BirfcoG+CxpqvMDgAs9YfcSCpLUlldpLJWS+B8bBb0e0vZrK4cj3QauygxqmSzJJv1BjbroQQ3JKF1UYQWrj+DoWuBVAcf+JSfU91BDgp3+1v0WH6Lvjv2/VJ0s+SgzhSAdBoOqum3DlwcB1X12oi3f+MNMh2nmR5ARMnDnrpERPX6mt17HRGlqsYABu96iKj1/D8fC6VvntDdGhaq2yi1acsoUapkoSQL1eBLryQb1Wk2yvqbq9+i79N/jI/qF0Obe38lX0NEjvbGK3kmeu4qHRymlTk07abI6uZRB8BHffXbBj5+0zm/kECXfD20h2S4eFZLot0L5mRbRpE9xAxBF7ksgYmhiBqdliDDMlJLEmSSIGuaICNk44AosyUEWQ6L2oBRJYHSeQJFQqULhkr1519VAJczcGOncf6u5JWBO3iBo52D3hQ9dhZGS9tmtHb0XxfOU68b+naghd9htFp6nnpbmaCmGa369SAZLcloSZh2cYzWXzGUDo99FiNzYjJLk2SWJLMkmdV0tJfeVjIr93JaAE8lmSXJLImS2ouSatuxgVnaw2M17PJdDo/1OlwoX38jcaHEhY1vcqp6GRfibVyIK3AhPjou7PZbcPC1vgWni/hI4v7O4/5LgbjnR2W/s7t4tHfqXGHwff2GywFblVqDKOHEW5VdedFPffXbZvPbusV3oq3Ko2+5yq1KScJdDwl3UoSyf6vy3TMNZ5SzY+9W1ilJyUpJVkqyUg2yUoZx9t3KelDUBoQqWYvOsxYSKF0wUKptxzZsOcOG5Wkcv05uWO4Hnl/YMEj4Ul+fA3hqEnhK4CmBZ8PA0zS1lgLPnK6XwFMCzxY0UALPjgLPbdjSHuDZMFPfcuApujCKeOHWD0JRpKOC3/0f</diagram></mxfile>

modules/sagemaker/sagemaker-templates-service-catalog/templates/finetune_llm_evaluation/__init__.py

@@ -0,0 +1 @@
+# Adding a comment here - empty files create issues with zipping https://github.com/aws/aws-cdk/issues/19012

modules/sagemaker/sagemaker-templates-service-catalog/templates/finetune_llm_evaluation/pipeline_constructs/build_pipeline_construct.py

@@ -0,0 +1,278 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from typing import Any
+
+import aws_cdk
+from aws_cdk import Aws
+from aws_cdk import aws_cloudwatch as cloudwatch
+from aws_cdk import aws_codebuild as codebuild
+from aws_cdk import aws_codecommit as codecommit
+from aws_cdk import aws_codepipeline as codepipeline
+from aws_cdk import aws_codepipeline_actions as codepipeline_actions
+from aws_cdk import aws_iam as iam
+from aws_cdk import aws_s3 as s3
+from aws_cdk import aws_s3_assets as s3_assets
+from constructs import Construct
+
+
+class BuildPipelineConstruct(Construct):
+    def __init__(
+        self,
+        scope: Construct,
+        construct_id: str,
+        project_name: str,
+        project_id: str,
+        model_package_group_name: str,
+        model_bucket: s3.IBucket,
+        pipeline_artifact_bucket: s3.IBucket,
+        repo_asset: s3_assets.Asset,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(scope, construct_id, **kwargs)
+
+        # Define resource name
+        sagemaker_pipeline_name = f"{project_name}-{project_id}"
+        sagemaker_pipeline_description = f"{project_name} Model Build Pipeline"
+
+        # Create source repo from seed bucket/key
+        build_app_repository = codecommit.Repository(
+            self,
+            "Build App Code Repo",
+            repository_name=f"{project_name}-{construct_id}",
+            code=codecommit.Code.from_asset(
+                asset=repo_asset,
+                branch="main",
+            ),
+        )
+        aws_cdk.Tags.of(build_app_repository).add("sagemaker:project-id", project_id)
+        aws_cdk.Tags.of(build_app_repository).add("sagemaker:project-name", project_name)
+
+        sagemaker_seedcode_bucket = s3.Bucket.from_bucket_name(
+            self,
+            "SageMaker Seedcode Bucket",
+            f"sagemaker-servicecatalog-seedcode-{Aws.REGION}",
+        )
+
+        codebuild_role = iam.Role(
+            self,
+            "CodeBuild Role",
+            assumed_by=iam.ServicePrincipal("codebuild.amazonaws.com"),
+            path="/service-role/",
+        )
+
+        sagemaker_execution_role = iam.Role(
+            self,
+            "SageMaker Execution Role",
+            assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com"),
+            path="/service-role/",
+        )
+
+        # Create a policy statement for SM and ECR pull
+        sagemaker_policy = iam.Policy(
+            self,
+            "SageMaker Policy",
+            document=iam.PolicyDocument(
+                statements=[
+                    iam.PolicyStatement(
+                        actions=[
+                            "logs:CreateLogGroup",
+                            "logs:CreateLogStream",
+                            "logs:PutLogEvents",
+                        ],
+                        resources=["*"],
+                    ),
+                    iam.PolicyStatement(
+                        actions=[
+                            "ecr:BatchCheckLayerAvailability",
+                            "ecr:BatchGetImage",
+                            "ecr:Describe*",
+                            "ecr:GetAuthorizationToken",
+                            "ecr:GetDownloadUrlForLayer",
+                        ],
+                        resources=["*"],
+                    ),
+                    iam.PolicyStatement(
+                        actions=[
+                            "kms:Encrypt",
+                            "kms:ReEncrypt*",
+                            "kms:GenerateDataKey*",
+                            "kms:Decrypt",
+                            "kms:DescribeKey",
+                        ],
+                        effect=iam.Effect.ALLOW,
+                        resources=[f"arn:{Aws.PARTITION}:kms:{Aws.REGION}:{Aws.ACCOUNT_ID}:key/*"],
+                    ),
+                ]
+            ),
+        )
+
+        cloudwatch.Metric.grant_put_metric_data(sagemaker_policy)
+        model_bucket.grant_read_write(sagemaker_policy)
+        sagemaker_seedcode_bucket.grant_read_write(sagemaker_policy)
+
+        sagemaker_execution_role.grant_pass_role(codebuild_role)
+        sagemaker_execution_role.grant_pass_role(sagemaker_execution_role)
+
+        # Attach the policy
+        sagemaker_policy.attach_to_role(sagemaker_execution_role)
+        sagemaker_policy.attach_to_role(codebuild_role)
+
+        # Grant extra permissions for the SageMaker role
+        sagemaker_execution_role.add_to_policy(
+            iam.PolicyStatement(
+                actions=[
+                    "sagemaker:CreateModel",
+                    "sagemaker:DeleteModel",
+                    "sagemaker:DescribeModel",
+                    "sagemaker:CreateProcessingJob",
+                    "sagemaker:DescribeProcessingJob",
+                    "sagemaker:StopProcessingJob",
+                    "sagemaker:CreateTrainingJob",
+                    "sagemaker:DescribeTrainingJob",
+                    "sagemaker:StopTrainingJob",
+                    "sagemaker:AddTags",
+                    "sagemaker:DeleteTags",
+                    "sagemaker:ListTags",
+                ],
+                resources=[
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model/*",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:processing-job/*",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:training-job/*",
+                ],
+            )
+        )
+        sagemaker_execution_role.add_to_policy(
+            iam.PolicyStatement(
+                actions=[
+                    "sagemaker:CreateModelPackageGroup",
+                    "sagemaker:DeleteModelPackageGroup",
+                    "sagemaker:DescribeModelPackageGroup",
+                    "sagemaker:CreateModelPackage",
+                    "sagemaker:DeleteModelPackage",
+                    "sagemaker:UpdateModelPackage",
+                    "sagemaker:DescribeModelPackage",
+                    "sagemaker:ListModelPackages",
+                    "sagemaker:AddTags",
+                    "sagemaker:DeleteTags",
+                    "sagemaker:ListTags",
+                ],
+                resources=[
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package-group/"
+                    f"{model_package_group_name}",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package/"
+                    f"{model_package_group_name}/*",
+                ],
+            ),
+        )
+
+        # Grant extra permissions for the CodeBuild role
+        codebuild_role.add_to_policy(
+            iam.PolicyStatement(
+                actions=[
+                    "sagemaker:DescribeModelPackage",
+                    "sagemaker:ListModelPackages",
+                    "sagemaker:UpdateModelPackage",
+                    "sagemaker:AddTags",
+                    "sagemaker:DeleteTags",
+                    "sagemaker:ListTags",
+                ],
+                resources=[
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:model-package/"
+                    f"{model_package_group_name}/*"
+                ],
+            )
+        )
+        codebuild_role.add_to_policy(
+            iam.PolicyStatement(
+                actions=[
+                    "sagemaker:CreatePipeline",
+                    "sagemaker:UpdatePipeline",
+                    "sagemaker:DeletePipeline",
+                    "sagemaker:StartPipelineExecution",
+                    "sagemaker:StopPipelineExecution",
+                    "sagemaker:DescribePipelineExecution",
+                    "sagemaker:ListPipelineExecutionSteps",
+                    "sagemaker:AddTags",
+                    "sagemaker:DeleteTags",
+                    "sagemaker:ListTags",
+                ],
+                resources=[
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
+                    f"{sagemaker_pipeline_name}",
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:pipeline/"
+                    f"{sagemaker_pipeline_name}/execution/*",
+                ],
+            ),
+        )
+        codebuild_role.add_to_policy(
+            iam.PolicyStatement(
+                actions=[
+                    "sagemaker:DescribeImageVersion",
+                ],
+                resources=[
+                    f"arn:{Aws.PARTITION}:sagemaker:{Aws.REGION}:{Aws.ACCOUNT_ID}:image-version/*",
+                ],
+            )
+        )
+
+        # Create the CodeBuild project
+        sm_pipeline_build = codebuild.PipelineProject(
+            self,
+            "SM Pipeline Build",
+            project_name=f"{project_name}-{construct_id}",
+            role=codebuild_role,  # figure out what actually this role would need
+            build_spec=codebuild.BuildSpec.from_source_filename("buildspec.yml"),
+            environment=codebuild.BuildEnvironment(
+                build_image=codebuild.LinuxBuildImage.STANDARD_5_0,
+                environment_variables={
+                    "SAGEMAKER_PROJECT_NAME": codebuild.BuildEnvironmentVariable(value=project_name),
+                    "SAGEMAKER_PROJECT_ID": codebuild.BuildEnvironmentVariable(value=project_id),
+                    "MODEL_PACKAGE_GROUP_NAME": codebuild.BuildEnvironmentVariable(value=model_package_group_name),
+                    "AWS_REGION": codebuild.BuildEnvironmentVariable(value=Aws.REGION),
+                    "SAGEMAKER_PIPELINE_NAME": codebuild.BuildEnvironmentVariable(
+                        value=sagemaker_pipeline_name,
+                    ),
+                    "SAGEMAKER_PIPELINE_DESCRIPTION": codebuild.BuildEnvironmentVariable(
+                        value=sagemaker_pipeline_description,
+                    ),
+                    "SAGEMAKER_PIPELINE_ROLE_ARN": codebuild.BuildEnvironmentVariable(
+                        value=sagemaker_execution_role.role_arn,
+                    ),
+                    "ARTIFACT_BUCKET": codebuild.BuildEnvironmentVariable(value=model_bucket.bucket_name),
+                    "ARTIFACT_BUCKET_KMS_ID": codebuild.BuildEnvironmentVariable(
+                        value=model_bucket.encryption_key.key_id  # type: ignore[union-attr]
+                    ),
+                },
+            ),
+        )
+
+        source_artifact = codepipeline.Artifact(artifact_name="GitSource")
+
+        build_pipeline = codepipeline.Pipeline(
+            self,
+            "Pipeline",
+            pipeline_name=f"{project_name}-{construct_id}",
+            artifact_bucket=pipeline_artifact_bucket,
+        )
+
+        # add a source stage
+        source_stage = build_pipeline.add_stage(stage_name="Source")
+        source_stage.add_action(
+            codepipeline_actions.CodeCommitSourceAction(
+                action_name="Source",
+                output=source_artifact,
+                repository=build_app_repository,
+                branch="main",
+            )
+        )
+
+        # add a build stage
+        build_stage = build_pipeline.add_stage(stage_name="Build")
+        build_stage.add_action(
+            codepipeline_actions.CodeBuildAction(
+                action_name="SMPipeline",
+                input=source_artifact,
+                project=sm_pipeline_build,
+            )
+        )