Skip to content

Commit

Permalink
Added a new Personas module. (#114)
Browse files Browse the repository at this point in the history
* Added a new Personas module.

* fix cdk issues

* Added a new Personas module.

* Added a new Personas module.

* Updated README.md

* Completed the checklist items

* mypy: add ignore for cdk default factory

* housekeeping

* changelog

* pr comments

* upd mypy conf
  • Loading branch information
rtdurga authored Jul 1, 2024
1 parent dd974dc commit 042e00b
Show file tree
Hide file tree
Showing 17 changed files with 1,136 additions and 25 deletions.
7 changes: 4 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### **Added**

- added `ray-on-eks`, and `manifests/ray-on-eks` manifests
- Added a `sagemaker-model-monitoring-module` module with an example of data quality, model quality, model bias, and model explainability monitoring of a SageMaker Endpoint.
- Added an option to enable data capture in the `sagemaker-endpoint-module`.
- added a `sagemaker-model-monitoring-module` module with an example of data quality, model quality, model bias, and model explainability monitoring of a SageMaker Endpoint
- added an option to enable data capture in the `sagemaker-endpoint-module`
- added a `personas` example module to deploy various roles required for an AI/ML project

### **Changed**
- remove explicit module manifest account/region mappings from `fmops-qna-rag`
Expand All @@ -23,7 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- fix inputs for `bedrock-finetuning` module not working
- add `retention-type` argument for the bucket in the `bedrock-finetuning` module
- fix broken dependencies for `examples/airflow-dags`
- Use `add_dependency` to avoid deprecation warnings from CDK.
- use `add_dependency` to avoid deprecation warnings from CDK.

## v1.2.0

Expand Down
39 changes: 22 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).

### SageMaker Modules

| Type | Description |
| Type | Description |
|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md) | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config |
| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md) | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group |
Expand All @@ -43,33 +43,38 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).

### FMOps/LLMOps Modules

| Type | Description |
|------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
| [SageMaker JumpStart Foundation Model Endpoint Module](modules/fmops/sagemaker-jumpstart-fm-endpoint/README.md) | Creates an endpoint for a SageMaker JumpStart Foundation Model. |
| [SageMaker Hugging Face Foundation Model Endpoint Module](modules/fmops/sagemaker-hugging-face-endpoint/README.md) | Creates an endpoint for a SageMaker Hugging Face Foundation Model. |
| [Amazon Bedrock Finetuning Module](modules/fmops/bedrock-finetuning/README.md) | Creates a pipeline that automatically triggers Amazon Bedrock Finetuning. |
| [AppSync Knowledge Base Ingestion and Question and Answering RAG Module](modules/fmops/qna-rag/README.md) | Creates an Graphql endpoint for ingestion of data and and use ingested as knowledge base for a Question and Answering model using RAG. |
| Type | Description |
|--------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| [SageMaker JumpStart Foundation Model Endpoint Module](modules/fmops/sagemaker-jumpstart-fm-endpoint/README.md) | Creates an endpoint for a SageMaker JumpStart Foundation Model. |
| [SageMaker Hugging Face Foundation Model Endpoint Module](modules/fmops/sagemaker-hugging-face-endpoint/README.md) | Creates an endpoint for a SageMaker Hugging Face Foundation Model. |
| [Amazon Bedrock Finetuning Module](modules/fmops/bedrock-finetuning/README.md) | Creates a pipeline that automatically triggers Amazon Bedrock Finetuning. |
| [AppSync Knowledge Base Ingestion and Question and Answering RAG Module](modules/fmops/qna-rag/README.md) | Creates an Graphql endpoint for ingestion of data and and use ingested as knowledge base for a Question and Answering model using RAG. |

### MWAA Modules

| Type | Description |
|-------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Example DAG for MLOps](modules/examples/airflow-dags/README.md) | Deploys a Sample DAG in MWAA demonstrating MLOPs and it is using MWAA module from IDF |
| Type | Description |
|-------------------------------------------------------------------------|------------------------------------------------------------------------------------------|
| [Example DAG for MLOps Module](modules/examples/airflow-dags/README.md) | Deploys a Sample DAG in MWAA demonstrating MLOPs and it is using MWAA module from IDF |


### EKS Modules

| Type | Description |
|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| [Ray on EKS](modules/eks/ray-on-eks/README.md) | Provisions Ray on EKS cluster using IDF EKS module, Ray Operator, and RayJob or RayCluster via Custom Resources. |
| Type | Description |
|-------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| [Ray on EKS Module](modules/eks/ray-on-eks/README.md) | Provisions Ray on EKS cluster using IDF EKS module, Ray Operator, and RayJob or RayCluster via Custom Resources. |

### Events Modules
### Example Modules

| Type | Description |
|-------------------------------------------------------------------|------------------------------------------------------------------------|
| [Event Bus Module](modules/examples/event-bus/README.md) | Creates an Amazon EventBridge Bus for cross-account events. |
| Type | Description |
|-----------------------------------------------------------|-------------------------------------------------------------------------------------|
| [Event Bus Module](modules/examples/event-bus/README.md) | Creates an Amazon EventBridge Bus for cross-account events. |
| [Personas Module](modules/examples/personas/README.md) | This module is an example that creates various roles required for an AI/ML project. |


### Industry Data Framework (IDF) Modules

The modules in this repository are compatible with [Industry Data Framework (IDF) Modules](https://github.com/awslabs/idf-modules) and can be used together within the same deployment. Refer to `examples/manifests` for examples.

### Autonomous Driving Data Framework (ADDF) Modules

The modules in this repository are compatible with [Autonomous Driving Data Framework (ADDF) Modules](https://github.com/awslabs/autonomous-driving-data-framework) and can be used together within the same deployment.
11 changes: 6 additions & 5 deletions examples/manifests/deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# This is an example manifest.
# Replace the parameters in referenced manifest groups with your values prior the deployment.
name: mlops
name: aiops
toolchainRegion: us-east-1
forceDependencyRedeploy: true
groups:
Expand All @@ -9,12 +9,13 @@ groups:
- name: sagemaker-endpoints
path: examples/manifests/sagemaker-endpoints-modules.yaml
- name: events
path: manifests/sagemaker-model-event-bus.yaml
path: examples/manifests/event-bus-modules.yaml
- name: registry
path: manifests/sagemaker-model-package-group-modules.yaml
path: examples/manifests/sagemaker-model-package-group-modules.yaml
- name: promote-models
path: manifests/sagemaker-model-package-promote-pipeline-modules.yaml

path: examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml
- name: personas
path: examples/manifests/personas-modules.yaml
targetAccountMappings:
- alias: primary
accountId:
Expand Down
File renamed without changes.
5 changes: 5 additions & 0 deletions examples/manifests/personas-modules.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
name: personas
path: modules/examples/personas
parameters:
- name: bucket-name
value: my-bucket
57 changes: 57 additions & 0 deletions modules/examples/personas/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Personas Module

## Description

This module is an example that deploys various roles that may be required for an AI/ML project, including:

- ML Engineer
- Data Engineer
- IT Lead
- Business Analyst
- MLOPs Engineer
- IT Auditor
- Model Risk Manager

The module creates separate roles with appropriate permissions and policies for each persona, ensuring proper segregation of duties and access control within the AI/ML project.

## Inputs/Outputs

### Input Parameters

#### Required

- `bucket_name`: S3 bucket name to add permissions for

## Sample manifest declaration

Create a manifest file under appropriate location, for example examples/manifests
```
name: personas
path: modules/examples/personas
parameters:
- name: bucket-name
value: my-bucket
```

### Outputs (module metadata):
- `MLEngineerRoleArn` - the arn of the Machine Learning Engineer Role
- `DataEngineerRoleArn` - the arn of the Data Engineer Role
- `ITLeadRoleArn` - the arn of the Machine IT Lead Role
- `BusinessAnalystRoleArn` - the arn of the Business Analyst Role
- `MLOpsEngineerRoleArn` - the arn of the Machine Learning Ops Engineer Role
- `ITAuditorRoleArn` - the arn of the IT Auditor Role
- `ModelRiskManagerRoleArn` - the arn of the Model Risk Manager Role

### Example Output:
```yaml
metadata: |
{
"MLEngineerRoleArn": "arn:aws:iam::<account_id>:role/MyStack-MLEngineerRole-<random_string>",
"DataEngineerRoleArn": "arn:aws:iam::<account_id>:role/MyStack-DataEngineerRole-<random_string>",
"ITLeadRoleArn": "arn:aws:iam::<account_id>:role/MyStack-ITLeadRole-<random_string>",
"BusinessAnalystRoleArn": "arn:aws:iam::<account_id>:role/MyStack-BusinessAnalystRole-<random_string>",
"MLOpsEngineerRoleArn": "arn:aws:iam::<account_id>:role/MyStack-MLOpsEngineerRole-<random_string>",
"ITAuditorRoleArn": "arn:aws:iam::<account_id>:role/MyStack-ITAuditorRole-<random_string>",
"ModelRiskManagerRoleArn": "arn:aws:iam::<account_id>:role/MyStack-ModelRiskManagerRole-<random_string>"
}
```
52 changes: 52 additions & 0 deletions modules/examples/personas/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

import aws_cdk
import cdk_nag

from settings import ApplicationSettings
from stack import Personas

app = aws_cdk.App()
# Load application settings from env vars.
app_settings = ApplicationSettings()


stack = Personas(
scope=app,
construct_id=app_settings.settings.app_prefix,
app_prefix=app_settings.settings.app_prefix,
bucket_name=app_settings.parameters.bucket_name,
env=aws_cdk.Environment(
account=app_settings.default.account,
region=app_settings.default.region,
),
)

aws_cdk.CfnOutput(
scope=stack,
id="metadata",
value=stack.to_json_string(
{
"MLEngineerRoleArn": stack.personas.ml_engineer_role.role_arn,
"DataEngineerRoleArn": stack.personas.data_engineer_role.role_arn,
"ITLeadRoleArn": stack.personas.it_lead_role.role_arn,
"BusinessAnalystRoleArn": stack.personas.business_analyst_role.role_arn,
"MLOpsEngineerRoleArn": stack.personas.mlops_engineer_role.role_arn,
"ITAuditorRoleArn": stack.personas.it_auditor_role.role_arn,
"ModelRiskManagerRoleArn": stack.personas.model_risk_manager_role.role_arn,
}
),
)

aws_cdk.Aspects.of(app).add(cdk_nag.AwsSolutionsChecks(log_ignores=True))

if app_settings.parameters.tags:
for tag_key, tag_value in app_settings.parameters.tags.items():
aws_cdk.Tags.of(app).add(tag_key, tag_value)

aws_cdk.Tags.of(app).add("SeedFarmerDeploymentName", app_settings.settings.deployment_name)
aws_cdk.Tags.of(app).add("SeedFarmerModuleName", app_settings.settings.module_name)
aws_cdk.Tags.of(app).add("SeedFarmerProjectName", app_settings.settings.project_name)

app.synth()
22 changes: 22 additions & 0 deletions modules/examples/personas/deployspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
publishGenericEnvVariables: true
deploy:
phases:
install:
commands:
- npm install -g [email protected]
- pip install -r requirements.txt
build:
commands:
- cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports_personas.json
# Export metadata
- seedfarmer metadata convert -f cdk-exports_personas.json || true
destroy:
phases:
install:
commands:
- npm install -g [email protected]
- pip install -r requirements.txt
build:
commands:
# execute the CDK
- cdk destroy --force --app "python app.py"
Loading

0 comments on commit 042e00b

Please sign in to comment.