Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding basic multi-account mlops usecase #18

Merged
merged 25 commits into from
Mar 7, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1dd168f
Added mlops multi account usecase
Feb 28, 2024
82a3c63
Linting
Feb 28, 2024
74268db
Merge pull request #1 from fotinosk/adding-multi-account-usecase
fotinosk Feb 28, 2024
2d6de07
Add app & stack.py with service catalog wrapper
kukushking Feb 28, 2024
ac8d4c2
Add sagemaker project templates manifest
kukushking Feb 28, 2024
1780b74
Linting
kukushking Feb 29, 2024
7b7e4c4
Housekeeping
kukushking Feb 29, 2024
e1ed95a
Update CHANGELOG.MD
kukushking Feb 29, 2024
f3c6746
Add basic tests
kukushking Feb 29, 2024
d9e1a79
Fix stack test case
kukushking Feb 29, 2024
6265cad
Ad README.MD
kukushking Feb 29, 2024
ffb301f
Update README.MD
kukushking Feb 29, 2024
66a2f2e
Merge branch 'main' into main
kukushking Mar 4, 2024
d000873
Update CHANGELOG.MD
kukushking Mar 4, 2024
a8382fc
Fix build pipeline & housekeeping
kukushking Mar 4, 2024
08b7874
Set test path
kukushking Mar 4, 2024
eed756e
add missing role permission
LeonLuttenberger Mar 4, 2024
5e40fb0
fix linting error
LeonLuttenberger Mar 4, 2024
c69828b
fix hardcoded regions and SSM reliance
LeonLuttenberger Mar 5, 2024
927cca7
temporarily remove vpc support for deploy pipeline
LeonLuttenberger Mar 6, 2024
489ce70
Merge branch 'main' into pr/fotinosk/18
LeonLuttenberger Mar 6, 2024
d4079fe
add stage name to endpoint name
LeonLuttenberger Mar 6, 2024
d9906ca
add stage name to model name
LeonLuttenberger Mar 6, 2024
6a75cc5
add stage name to endpoint config name
LeonLuttenberger Mar 6, 2024
e590c74
add link to enterprise mlops framework readme
kukushking Mar 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### **Added**

- added `sagemaker-templates` module with `multi_account_basic` project template
- added `sagemaker-jumpstart-fm-endpoint` module
- added RDS persistence layer to MLFlow modules
- added `mlflow-image` and `mlflow-fargate` modules
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,12 @@ All modules in this repository adhere to the module structure defined in the the

### SageMaker Modules

| Type | Description |
|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md) | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group |
| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md) | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config |
| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md) | Creates SageMaker Notebook Instances |
| Type | Description |
|---------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md) | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config |
| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md) | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group |
| [SageMaker Project Templates Module](modules/sagemaker/sagemaker-templates/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog |
| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md) | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace |

### Mlflow Modules

Expand Down
2 changes: 2 additions & 0 deletions manifests/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ groups:
path: manifests/storage-modules.yaml
- name: sagemaker-studio
path: manifests/sagemaker-studio-modules.yaml
- name: sagemaker-templates
path: manifests/sagemaker-templates-modules.yaml
- name: images
path: manifests/images-modules.yaml
- name: mlflow
Expand Down
10 changes: 10 additions & 0 deletions manifests/sagemaker-templates-modules.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: templates
path: modules/sagemaker/sagemaker-templates
targetAccount: primary
parameters:
- name: portfolio-access-role-arn
valueFrom:
moduleMetadata:
group: sagemaker-studio
name: studio
key: LeadDataScientistRoleArn
44 changes: 44 additions & 0 deletions modules/sagemaker/sagemaker-templates/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# SageMaker Project Templates

This module creates organizational SageMaker Project Templates.

The templates are registered in Service Catalog and available via SageMaker Studio Classic.

### Architecture

![SageMaker Templates Module Architecture](docs/_static/sagemaker-templates-module-architecture.png "SageMaker Templates Module Architecture")

## Inputs and outputs:
### Required inputs:
- `portfolio-access-role-arn` - the ARN of the IAM Role used to access the Service Catalog Portfolio or SageMaker projects

### Optional Inputs:
- `portfolio-name` - name of the Service Catalog Portfolio
- `portfolio-owner` - owner of the Service Catalog Portfolio

### Sample manifest declaration

```yaml
name: templates
path: modules/sagemaker/sagemaker-templates
targetAccount: primary
parameters:
- name: portfolio-access-role-arn
valueFrom:
moduleMetadata:
group: sagemaker-studio
name: studio
key: LeadDataScientistRoleArn
```

### Outputs (module metadata):
- `ServiceCatalogPortfolioName` - the name of the Service Catalog Portfolio
- `ServiceCatalogPortfolioOwner` - the owner of the Service Catalog Portfolio

### Example Output:
```yaml
{
"ServiceCatalogPortfolioName": "MLOps SageMaker Project Templates",
"ServiceCatalogPortfolioOwner": "administrator"
}
```
56 changes: 56 additions & 0 deletions modules/sagemaker/sagemaker-templates/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

import os

import aws_cdk

from stack import ServiceCatalogStack

project_name = os.getenv("SEEDFARMER_PROJECT_NAME", "")
deployment_name = os.getenv("SEEDFARMER_DEPLOYMENT_NAME", "")
module_name = os.getenv("SEEDFARMER_MODULE_NAME", "")
app_prefix = f"{project_name}-{deployment_name}-{module_name}"

DEFAULT_PORTFOLIO_NAME = "MLOps SageMaker Project Templates"
DEFAULT_PORTFOLIO_OWNER = "administrator"


def _param(name: str) -> str:
return f"SEEDFARMER_PARAMETER_{name}"


environment = aws_cdk.Environment(
account=os.environ["CDK_DEFAULT_ACCOUNT"],
region=os.environ["CDK_DEFAULT_REGION"],
)

portfolio_name = os.getenv(_param("PORTFOLIO_NAME"), DEFAULT_PORTFOLIO_NAME)
portfolio_owner = os.getenv(_param("PORTFOLIO_OWNER"), DEFAULT_PORTFOLIO_OWNER)
portfolio_access_role_arn = os.getenv(_param("PORTFOLIO_ACCESS_ROLE_ARN"))

if not portfolio_access_role_arn:
raise ValueError("Missing input parameter portfolio-access-role-arn")

app = aws_cdk.App()
stack = ServiceCatalogStack(
app,
app_prefix,
portfolio_name=portfolio_name,
portfolio_owner=portfolio_owner,
portfolio_access_role_arn=portfolio_access_role_arn,
)


aws_cdk.CfnOutput(
scope=stack,
id="metadata",
value=stack.to_json_string(
{
"ServiceCatalogPortfolioName": stack.portfolio_name,
"ServiceCatalogPortfolioOwner": stack.portfolio_owner,
}
),
)

app.synth()
3 changes: 3 additions & 0 deletions modules/sagemaker/sagemaker-templates/coverage.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[run]
omit =
tests/*
23 changes: 23 additions & 0 deletions modules/sagemaker/sagemaker-templates/deployspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
publishGenericEnvVariables: true
deploy:
phases:
install:
commands:
- env
# Install whatever additional build libraries
- npm install -g [email protected]
- pip install -r requirements.txt
build:
commands:
- cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports.json
destroy:
phases:
install:
commands:
# Install whatever additional build libraries
- npm install -g [email protected]
- pip install -r requirements.txt
build:
commands:
# execute the CDK
- cdk destroy --force --app "python app.py"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
FROM public.ecr.aws/docker/library/alpine:latest

RUN apk --no-cache add zip make
38 changes: 38 additions & 0 deletions modules/sagemaker/sagemaker-templates/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
[tool.black]
line-length = 120
target-version = ["py36", "py37", "py38"]
exclude = '''
/(
\.eggs
| \.git
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| \.env
| _build
| buck-out
| build
| dist
| codeseeder.out
)/
'''

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
line_length = 120
py_version = 36
skip_gitignore = false

[tool.pytest.ini_options]
addopts = "-v --cov=. --cov-report term --cov-config=coverage.ini --cov-fail-under=80"
pythonpath = [
"."
]
testpaths = [
"tests"
]
3 changes: 3 additions & 0 deletions modules/sagemaker/sagemaker-templates/requirements.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
aws-cdk-lib==2.130.0
cdk-nag==2.28.27
boto3==1.34.35
84 changes: 84 additions & 0 deletions modules/sagemaker/sagemaker-templates/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --output-file=requirements.txt requirements.in
#
attrs==23.2.0
# via
# cattrs
# jsii
aws-cdk-asset-awscli-v1==2.2.202
# via aws-cdk-lib
aws-cdk-asset-kubectl-v20==2.1.2
# via aws-cdk-lib
aws-cdk-asset-node-proxy-agent-v6==2.0.1
# via aws-cdk-lib
aws-cdk-lib==2.130.0
# via
# -r requirements.in
# cdk-nag
boto3==1.34.35
# via -r requirements.in
botocore==1.34.51
# via
# boto3
# s3transfer
cattrs==23.2.3
# via jsii
cdk-nag==2.28.27
# via -r requirements.in
constructs==10.3.0
# via
# aws-cdk-lib
# cdk-nag
exceptiongroup==1.2.0
# via cattrs
importlib-resources==6.1.2
# via jsii
jmespath==1.0.1
# via
# boto3
# botocore
jsii==1.94.0
# via
# aws-cdk-asset-awscli-v1
# aws-cdk-asset-kubectl-v20
# aws-cdk-asset-node-proxy-agent-v6
# aws-cdk-lib
# cdk-nag
# constructs
publication==0.0.3
# via
# aws-cdk-asset-awscli-v1
# aws-cdk-asset-kubectl-v20
# aws-cdk-asset-node-proxy-agent-v6
# aws-cdk-lib
# cdk-nag
# constructs
# jsii
python-dateutil==2.8.2
# via
# botocore
# jsii
s3transfer==0.10.0
# via boto3
six==1.16.0
# via python-dateutil
typeguard==2.13.3
# via
# aws-cdk-asset-awscli-v1
# aws-cdk-asset-kubectl-v20
# aws-cdk-asset-node-proxy-agent-v6
# aws-cdk-lib
# cdk-nag
# constructs
# jsii
typing-extensions==4.10.0
# via
# cattrs
# jsii
urllib3==1.26.18
# via botocore
zipp==3.17.0
# via importlib-resources
28 changes: 28 additions & 0 deletions modules/sagemaker/sagemaker-templates/setup.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
[metadata]
license_files =
LICENSE
NOTICE
VERSION

[flake8]
max-line-length = 120
extend-ignore = E203, W503
exclude =
.git,
__pycache__,
docs/source/conf.py,
old,
build,
dist,
.venv,
codeseeder.out,
bundle

[mypy]
python_version = 3.7
strict = True
ignore_missing_imports = True
allow_untyped_decorators = True
exclude =
codeseeder.out/|example/|tests/
warn_unused_ignores = False
Loading
Loading