feat: add SageMaker model package promote modules (#61)

* Add model pkg group event module * Refactoring * Add sm model event bus module * Refactoring * Add sm model pkg promote module * Fix linging * Fix linting * Upd changelog * Fix typos * Fix linting problems * Fix format * Move event bus module to another location * Rename event bus module * Fix event bus module location * Upd readme with new modules * Fix module path --------- Co-authored-by: kukushking <[email protected]>
awslabs · May 3, 2024 · 9959455 · 9959455
1 parent b474d12
commit 9959455
Show file tree

Hide file tree

Showing 59 changed files with 2,740 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -18,6 +18,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - added batch inference project template to `sagemaker-templates-service-catalog` module
 - added EFS removal policy to `mlflow-fargate` module
 - added `mwaa` module with example dag which demonstrates the MLOps in Airflow
+- added `sagemaker-model-event-bus` module.
+- added `sagemaker-model-package-group` module.
+- added `sagemaker-model-package-promote-pipeline` module.
 - added `sagemaker-hugging-face-endpoint` module
 - added `hf_import_models` template to import hugging face models
 

diff --git a/README.md b/README.md
@@ -23,14 +23,15 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).
 
 ### SageMaker Modules
 
-| Type                                                                                                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| Type                                                                                                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 |---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)                                                   | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md)                                               | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | [SageMaker Project Templates via Service Catalog Module](modules/sagemaker/sagemaker-templates-service-catalog/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog. Available templates:<br/> - [Train a model on Abalone dataset using XGBoost](modules/sagemaker/sagemaker-templates-service-catalog/README.md#train-a-model-on-abalone-dataset-with-xgboost-template)<br/>- [Perform batch inference](modules/sagemaker/sagemaker-templates-service-catalog/README.md#batch-inference-template)<br/>- [Multi-account model deployment](modules/sagemaker/sagemaker-templates-service-catalog/README.md#multi-account-model-deployment-template) <br/>- [HuggingFace model import template](modules/sagemaker/sagemaker-templates-service-catalog/README.md#huggingface-model-import-template) |
 | [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md)                                      | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
 | [SageMaker Custom Kernel Module](modules/sagemaker/sagemaker-custom-kernel/README.md)                                     | Builds custom kernel for SageMaker Studio from a Dockerfile                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
-
+| [SageMaker Model Package Group Module](modules/sagemaker/sagemaker-model-package-group/README.md)                         | Creates a SageMaker Model Package Group to register and version SageMaker Machine Learning (ML) models and setups an Amazon EventBridge Rule to send model package group state change events to an Amazon EventBridge Bus                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| [SageMaker Model Package Promote Pipeline Module](modules/sagemaker/sagemaker-model-package-promote-pipeline/README.md)   | Deploy a Pipeline to promote SageMaker Model Packages in a multi-account setup. The pipeline can be triggered through an EventBridge rule in reaction of a SageMaker Model Package Group state event change (Approved/Rejected). Once the pipeline is triggered, it will promote the latest approved model package, if one is found.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 
 ### Mlflow Modules
 
@@ -56,3 +57,9 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).
 ### Industry Data Framework (IDF) Modules
 
 The modules in this repository are compatible with [Industry Data Framework (IDF) Modules](https://github.com/awslabs/idf-modules) and can be used together within the same deployment. Refer to `examples/manifests` for examples.
+
+### Events Modules
+
+| Type                                                                    | Description                                                                                                                                                                                       |
+|-------------------------------------------------------------------------|------------------------------------------------------------------------|
+|  [Event Bus Module](modules/examples/events/event-bus/README.md)        |  Creates an Amazon EventBridge Bus for cross-account events.           |
diff --git a/examples/manifests/deployment.yaml b/examples/manifests/deployment.yaml
@@ -8,6 +8,12 @@ groups:
     path: examples/manifests/networking-modules.yaml
   - name: sagemaker-endpoints
     path: examples/manifests/sagemaker-endpoints-modules.yaml
+  - name: events
+    path: manifests/sagemaker-model-event-bus.yaml
+  - name: registry
+    path: manifests/sagemaker-model-package-group-modules.yaml
+  - name: promote-models
+    path: manifests/sagemaker-model-package-promote-pipeline-modules.yaml
 targetAccountMappings:
   - alias: primary
     accountId:
@@ -17,3 +23,10 @@ targetAccountMappings:
     regionMappings:
       - region: us-east-1
         default: true
+  - alias: tooling
+    accountId:
+      valueFrom:
+        envVariable: TOOLING_ACCOUNT
+    regionMappings:
+      - region: us-east-1
+        default: true
diff --git a/examples/manifests/event-bus.yaml b/examples/manifests/event-bus.yaml
@@ -0,0 +1,10 @@
+name: event-bus
+path: modules/examples/events/event-bus
+targetAccount: tooling
+parameters:
+  - name: event_bus_name
+    value: mlops-bus
+  - name: source_accounts
+    value: '["123123123123"]' # Accounts that must have permissions to put events (source accounts)
+  - name: tags
+    value: '{"test": "test"}'
diff --git a/examples/manifests/sagemaker-model-package-group-modules.yaml b/examples/manifests/sagemaker-model-package-group-modules.yaml
@@ -0,0 +1,33 @@
+name: source-model-package-group
+path: modules/sagemaker/sagemaker-model-package-group
+targetAccount: primary
+parameters:
+  - name: model_package_group_name
+    value: mlops-test-model-group-source
+  - name: target_event_bus_arn
+    valueFrom:
+      moduleMetadata:
+        group: events
+        name: event-bus
+        key: EventBusArn
+  - name: target_account_ids
+    value: '["444333222555"]' # Accounts that must have read-only permissions on the model pkg group
+  - name: sagemaker_project_id
+    value: 123123
+  - name: sagemaker_project_name
+    value: test
+---
+name: target-model-package-group
+path: modules/sagemaker/sagemaker-model-package-group
+targetAccount: tooling
+parameters:
+  - name: model_package_group_name
+    value: mlops-test-model-group-tooling
+  - name: model_package_group_description
+    value: Test model package group module - Target
+  - name: target_account_ids
+    value: '["111222333444"]' # Accounts that must have read-only permissions on the model pkg group
+  - name: sagemaker_project_id
+    value: 123123
+  - name: sagemaker_project_name
+    value: test
diff --git a/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml b/examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml
@@ -0,0 +1,24 @@
+name: rappi-b2
+path: modules/sagemaker/sagemaker-model-package-promote-pipeline
+targetAccount: tooling
+parameters:
+  - name: source_model_package_group_arn
+    valueFrom:
+      moduleMetadata:
+        group: registry
+        name: source-model-package-group
+        key: SagemakerModelPackageGroupArn
+  - name: target_bucket_name
+    value: my-bucket-name
+  - name: event_bus_name
+    valueFrom:
+      moduleMetadata:
+        group: events
+        name: event-bus
+        key: EventBusName
+  - name: target_model_package_group_name
+    valueFrom:
+      moduleMetadata:
+        group: registry
+        name: target-model-package-group
+        key: SagemakerModelPackageGroupName
diff --git a/modules/examples/events/event-bus/README.md b/modules/examples/events/event-bus/README.md
@@ -0,0 +1,51 @@
+# Event Bus
+
+## Description
+
+This module creates an Amazon EventBridge Bus for cross-account events.
+
+### Architecture
+
+![Event Bus Architecture](docs/_static/architecture.drawio.png "Event Bus Architecture")
+
+## Inputs/Outputs
+
+### Input Paramenters
+
+#### Required
+
+- `event_bus_name`: EventBridge Bus name.
+
+#### Optional
+
+- `source_accounts`: A list of account ids which shall have write access to the event bridge bus. Defaults None.
+- `tags`: A dictionary of tags. Defaults None.
+
+### Sample manifest declaration
+
+```yaml
+name: event-bus
+path: modules/examples/events/event-bus
+targetAccount: primary
+parameters:
+  - name: event_bus_name
+    value: mlops-bus
+  - name: source_accounts
+    value: '["111222333444", "555666777888"]'
+  - name: tags
+    value: '{"key": "value"}'
+```
+
+### Module Metadata Outputs
+
+- `EventBusArn`: the EventBridge bus ARN.
+- `EventBusName`: the EventBridge bus name.
+
+#### Output Example
+
+```json
+{
+  "EventBusArn": "arn:aws:events:xx-xxxx-x:xxxxxxxxxx:event-bus/mlops-bus",
+  "EventBusName": "mlops-bus",
+}
+```
diff --git a/modules/examples/events/event-bus/app.py b/modules/examples/events/event-bus/app.py
@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+"""Create a Event Bus Stack."""
+
+import aws_cdk as cdk
+
+from event_bus.settings import ApplicationSettings
+from event_bus.stack import EventBusStack
+
+# Load application settings from env vars.
+app_settings = ApplicationSettings()
+
+env = cdk.Environment(
+    account=app_settings.default.account,
+    region=app_settings.default.region,
+)
+
+app = cdk.App()
+
+stack = EventBusStack(
+    scope=app,
+    construct_id=app_settings.settings.app_prefix,
+    env=env,
+    **app_settings.parameters.model_dump(),
+)
+
+app.synth()
diff --git a/modules/examples/events/event-bus/deployspec.yaml b/modules/examples/events/event-bus/deployspec.yaml
@@ -0,0 +1,26 @@
+publishGenericEnvVariables: true
+deploy:
+  phases:
+    install:
+      commands:
+      - env
+      # Install whatever additional build libraries
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    build:
+      commands:
+        # execute the CDK
+      - cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports.json
+      # Export metadata
+      - seedfarmer metadata convert -f cdk-exports.json || true
+destroy:
+  phases:
+    install:
+      commands:
+      # Install whatever additional build libraries
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    build:
+      commands:
+      # execute the CDK
+      - cdk destroy --force --app "python app.py"
diff --git a/modules/examples/events/event-bus/docs/_static/architecture.drawio b/modules/examples/events/event-bus/docs/_static/architecture.drawio
@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2024-03-23T13:09:48.468Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/20.2.3 Chrome/102.0.5005.167 Electron/19.0.11 Safari/537.36" etag="0olYNljB-bfJTNhMpkqE" version="20.2.3" type="device"><diagram id="nOkpBZcZqbjnXywVXBLF" name="Page-1">7Vpbc9o4FP41PIbxHfKITUi3Tbtp2dlO9oVRbGHcCIvKMpf++pVsCWNJBHZC07CFZAbr+Ojio+9837Fxx43m61sCFrOPOIGo41jJuuMOO45j216ffXHLprZ4Pbc2pCRLhFNjGGc/oDBawlpmCSxajhRjRLNF2xjjPIcxbdkAIXjVdpti1J51AVKoGcYxQLr1a5bQWW3t+1ZjfwezdCZnti1xZg6kszAUM5Dg1Y7Jvem4EcGY1kfzdQQRD56MS91vtOfsdmEE5vSYDt8+p/H9YDkduGH/LkjourQersQoS4BKccGDr2NmGMQxLtm49crpRoZjgbOcViH1Q/bPZoysjs/ORLzVdXzFoLZ7bYOtt/gYbYPa7rUNtjq8rcxvqwvcMWit1vCWMr+1s0D274a4pCjLYbQFn8WMKQFJxjYlwggTZstxzqIXzugcsZbNDlezjMLxAsQ8qiuWOMw2xTkV8Lcd2RaB56MygFPA5iJijGonILlZwnpDah+EwKLIHre9CIxLUmRL+AUW9eDcyqC44MfzdcqztgtWhddNCS4X1fL/YHMZz07Y4SRGuEwmAFE+ECX4CcoL7Tgu+xtx+IXTDCElAEtIaMYya4CylI9PMZ8OiBaC02pEFpUsT++q1tC1RCRMUySgmMFEXJKeDRLabFa43jGJ7LiFeA4p2TAXcdaVSS2oSjZXTd47MrtnOznvygwHgmvS7dBNOrIDkZF7snOJ4M37+O774Gmw/vihWP91i65cLTu1hDTEWduUaOgEQbB3B1SkKfH2B0HUD7Yh1uLZjvp+ltkbd0cJu6+F3Q4MYXds/yeF3TOEPUACnXkr/sH3kvN3yEJCrwSUB8xDoLlxYEcp/67yNWRSx2THsaoW+w7LQk7BVlzPUvtrG148QRrPxFYZ6dhIySZaNlKzTs8tt4owDTOoRpOtpxtt3U1yrG402UyCova2Db1tpfd+Ot9HPyrNs3OjkTfqhzvnhhkjYJpVdJpjwvHcYkbWJ4xs1w9MaTutPirRyRy+A48Q3eMiE8PPsyRBJprdnlCYdkePDmkPKBZ1PKbZmi/ELCAEFrgkMazlg0lPYRISyCH/WCeAGyLlKkid34oMOL+UfRzfQD4n4J74A3g/nz2FydPD6mMwgJMp+mEoyCRHPHKOeI4MXoReoR0qOC2r7w01cArno3D5iCnF84PyH0NezbwQlmpNZO+D6g4IJwmcghLRSWWbsChPJJD5iJBkbGMhGdeAZAN5xmrj2er66BKk53Wv+y082tdW19r52J6GT595uIayxPtJ8hhoEP0iA+ZY9xhl8eb1gTqy+rbOomcPVIa/eVYUbIVFw4qtkvf0jNj3uvw2toFcG5C2Xha7drd3rSPQO0FdvLaG9/Ok9zn6B7yL0J8Pn9LobwNJjiX+lNtXufjLLezlFvY3uIW1vYO3sHbPcC+1fWp1+lS19hU0l1LmdKWMoZLerqINR0uTkaNLnGep+GiIBn4Loa6OUM8A0FNIibGY6R0vJfrDl4uUXKTkDUvJgdLwv0nJ9hncq2iJMVdt56IlZ6wlh+5U9tLzuWjJ9fFa8umiJRct+W21xOn9ei3Rf1u7aMn/W0uuz0lL5HPeHSTCJIVyb/hvSTjFOUA3jTVkTJInW7g0PneYA6Haq2+Q0o3YcVBS3IZDPSdMtHditGcS8gn5wQSjgKRQ9N3/i4e+LQQiQBm9tmY1hbnqOiAEbHYchJo2I99zQ7PbrttmJFd9eeaAv2SwZovrFTQbvr2UF2DAP3cMOG8aA8pPf/Kds70YUP2D18BAcD4YePYB4BvFgJLXsuA4mgf8F2GANZsXAGv35jVK9+Zf</diagram></mxfile>
diff --git a/modules/examples/events/event-bus/docs/_static/architecture.drawio.png b/modules/examples/events/event-bus/docs/_static/architecture.drawio.png
diff --git a/modules/examples/events/event-bus/event_bus/__init__.py b/modules/examples/events/event-bus/event_bus/__init__.py
diff --git a/modules/examples/events/event-bus/event_bus/settings.py b/modules/examples/events/event-bus/event_bus/settings.py
@@ -0,0 +1,73 @@
+"""Defines the stack settings."""
+
+from abc import ABC
+from typing import Dict, List, Optional
+
+from pydantic import Field, computed_field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class CdkBaseSettings(BaseSettings, ABC):
+    """Defines common configuration for settings."""
+
+    model_config = SettingsConfigDict(
+        case_sensitive=False,
+        env_nested_delimiter="__",
+        protected_namespaces=(),
+        extra="ignore",
+        populate_by_name=True,
+    )
+
+
+class SeedFarmerParameters(CdkBaseSettings):
+    """Seedfarmer Parameters.
+
+    These parameters are required for the module stack.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="SEEDFARMER_PARAMETER_")
+
+    event_bus_name: str
+
+    source_accounts: Optional[List[str]] = Field(default=None)
+    tags: Optional[Dict[str, str]] = Field(default=None)
+
+
+class SeedFarmerSettings(CdkBaseSettings):
+    """Seedfarmer Settings.
+
+    These parameters comes from seedfarmer by default.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="SEEDFARMER_")
+
+    project_name: str = Field(default="")
+    deployment_name: str = Field(default="")
+    module_name: str = Field(default="")
+
+    @computed_field  # type: ignore
+    @property
+    def app_prefix(self) -> str:
+        """Application prefix."""
+        prefix = "-".join([self.project_name, self.deployment_name, self.module_name])
+        return prefix
+
+
+class CdkDefaultSettings(CdkBaseSettings):
+    """CDK Default Settings.
+
+    These parameters comes from AWS CDK by default.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="CDK_DEFAULT_")
+
+    account: str
+    region: str
+
+
+class ApplicationSettings(CdkBaseSettings):
+    """Application settings."""
+
+    settings: SeedFarmerSettings = Field(default_factory=SeedFarmerSettings)
+    parameters: SeedFarmerParameters = Field(default_factory=SeedFarmerParameters)
+    default: CdkDefaultSettings = Field(default_factory=CdkDefaultSettings)
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		<mxfile host="Electron" modified="2024-03-23T13:09:48.468Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/20.2.3 Chrome/102.0.5005.167 Electron/19.0.11 Safari/537.36" etag="0olYNljB-bfJTNhMpkqE" version="20.2.3" type="device"><diagram id="nOkpBZcZqbjnXywVXBLF" name="Page-1">7Vpbc9o4FP41PIbxHfKITUi3Tbtp2dlO9oVRbGHcCIvKMpf++pVsCWNJBHZC07CFZAbr+Ojio+9837Fxx43m61sCFrOPOIGo41jJuuMOO45j216ffXHLprZ4Pbc2pCRLhFNjGGc/oDBawlpmCSxajhRjRLNF2xjjPIcxbdkAIXjVdpti1J51AVKoGcYxQLr1a5bQWW3t+1ZjfwezdCZnti1xZg6kszAUM5Dg1Y7Jvem4EcGY1kfzdQQRD56MS91vtOfsdmEE5vSYDt8+p/H9YDkduGH/LkjourQersQoS4BKccGDr2NmGMQxLtm49crpRoZjgbOcViH1Q/bPZoysjs/ORLzVdXzFoLZ7bYOtt/gYbYPa7rUNtjq8rcxvqwvcMWit1vCWMr+1s0D274a4pCjLYbQFn8WMKQFJxjYlwggTZstxzqIXzugcsZbNDlezjMLxAsQ8qiuWOMw2xTkV8Lcd2RaB56MygFPA5iJijGonILlZwnpDah+EwKLIHre9CIxLUmRL+AUW9eDcyqC44MfzdcqztgtWhddNCS4X1fL/YHMZz07Y4SRGuEwmAFE+ECX4CcoL7Tgu+xtx+IXTDCElAEtIaMYya4CylI9PMZ8OiBaC02pEFpUsT++q1tC1RCRMUySgmMFEXJKeDRLabFa43jGJ7LiFeA4p2TAXcdaVSS2oSjZXTd47MrtnOznvygwHgmvS7dBNOrIDkZF7snOJ4M37+O774Gmw/vihWP91i65cLTu1hDTEWduUaOgEQbB3B1SkKfH2B0HUD7Yh1uLZjvp+ltkbd0cJu6+F3Q4MYXds/yeF3TOEPUACnXkr/sH3kvN3yEJCrwSUB8xDoLlxYEcp/67yNWRSx2THsaoW+w7LQk7BVlzPUvtrG148QRrPxFYZ6dhIySZaNlKzTs8tt4owDTOoRpOtpxtt3U1yrG402UyCova2Db1tpfd+Ot9HPyrNs3OjkTfqhzvnhhkjYJpVdJpjwvHcYkbWJ4xs1w9MaTutPirRyRy+A48Q3eMiE8PPsyRBJprdnlCYdkePDmkPKBZ1PKbZmi/ELCAEFrgkMazlg0lPYRISyCH/WCeAGyLlKkid34oMOL+UfRzfQD4n4J74A3g/nz2FydPD6mMwgJMp+mEoyCRHPHKOeI4MXoReoR0qOC2r7w01cArno3D5iCnF84PyH0NezbwQlmpNZO+D6g4IJwmcghLRSWWbsChPJJD5iJBkbGMhGdeAZAN5xmrj2er66BKk53Wv+y082tdW19r52J6GT595uIayxPtJ8hhoEP0iA+ZY9xhl8eb1gTqy+rbOomcPVIa/eVYUbIVFw4qtkvf0jNj3uvw2toFcG5C2Xha7drd3rSPQO0FdvLaG9/Ok9zn6B7yL0J8Pn9LobwNJjiX+lNtXufjLLezlFvY3uIW1vYO3sHbPcC+1fWp1+lS19hU0l1LmdKWMoZLerqINR0uTkaNLnGep+GiIBn4Loa6OUM8A0FNIibGY6R0vJfrDl4uUXKTkDUvJgdLwv0nJ9hncq2iJMVdt56IlZ6wlh+5U9tLzuWjJ9fFa8umiJRct+W21xOn9ei3Rf1u7aMn/W0uuz0lL5HPeHSTCJIVyb/hvSTjFOUA3jTVkTJInW7g0PneYA6Haq2+Q0o3YcVBS3IZDPSdMtHditGcS8gn5wQSjgKRQ9N3/i4e+LQQiQBm9tmY1hbnqOiAEbHYchJo2I99zQ7PbrttmJFd9eeaAv2SwZovrFTQbvr2UF2DAP3cMOG8aA8pPf/Kds70YUP2D18BAcD4YePYB4BvFgJLXsuA4mgf8F2GANZsXAGv35jVK9+Zf</diagram></mxfile>