Merge pull request #5 from kukushking/feat/mlflow

feat: add `mlflow` modules

CHANGELOG.md

@@ -11,11 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### **Added**
 
+- added `mlflow-image` and `mlflow-fargate` modules
 - added `sagemaker-studio` module
 - added `sagemaker-endpoint` module
 
 ### **Changed**
 
 ### **Removed**
-
-=======

README.md

@@ -8,7 +8,7 @@ The modules in this repository are decoupled from each other and can be aggregat
 
 The modules in this repository are / must be generic for reuse without affiliation to any one particular project in Machine Learning Operations domain.
 
-All modules in this repository adhere to the module strutucture defined in the the [SeedFarmer Guide](https://seed-farmer.readthedocs.io/en/latest)
+All modules in this repository adhere to the module structure defined in the the [SeedFarmer Guide](https://seed-farmer.readthedocs.io/en/latest)
 
 - [Project Structure](https://seed-farmer.readthedocs.io/en/latest/project_development.html)
 - [Module Development](https://seed-farmer.readthedocs.io/en/latest/module_development.html)
@@ -24,6 +24,13 @@ All modules in this repository adhere to the module strutucture defined in the t
 | [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)     | Creates SageMaker Studio Domain.                |
 
 
+### Mlflow Modules
+
+| Type                                                                    | Description                     |
+|-------------------------------------------------------------------------|---------------------------------|
+| [Mlflow Image Module](modules/mlflow/mlflow-image/README.md)            | Creates Mlflow container image. |
+| [Mlflow on AWS Fargate Module](modules/mlflow/mlflow-fargate/README.md) | Runs Mlflow on AWS Fargate.     |
+
 ### Industry Data Framework (IDF) Modules
 
 The modules in this repository are compatible with [Industry Data Framework (IDF) Modules](https://github.com/awslabs/idf-modules) and can be used together within the same deployment. Refer to `examples/manifests` for examples.

manifests/deployment.yaml

@@ -4,8 +4,14 @@ forceDependencyRedeploy: true
 groups:
   - name: networking
     path: manifests/networking-modules.yaml
+  - name: storage
+    path: manifests/storage-modules.yaml
   - name: sagemaker-studio
     path: manifests/sagemaker-studio-modules.yaml
+  - name: images
+    path: manifests/images-modules.yaml
+  - name: mlflow
+    path: manifests/mlflow-modules.yaml
 targetAccountMappings:
   - alias: primary
     accountId:

manifests/images-modules.yaml

@@ -0,0 +1,9 @@
+name: mlflow-image
+path: modules/mlflow/mlflow-image
+parameters:
+  - name: ecr-repository-name
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: ecr-mlflow
+        key: EcrRepositoryName

manifests/mlflow-modules.yaml

@@ -0,0 +1,27 @@
+name: mlflow-fargate
+path: modules/mlflow/mlflow-fargate
+parameters:
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId
+  - name: subnet-ids
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: PrivateSubnetIds
+  - name: ecr-repository-name
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: ecr-mlflow
+        key: EcrRepositoryName
+  - name: artifacts-bucket-name
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: buckets
+        key: ArtifactsBucketName

manifests/sagemaker-studio-modules.yaml

@@ -3,7 +3,7 @@ path: modules/sagemaker/sagemaker-studio
 targetAccount: primary
 parameters:
   - name: studio_domain_name
-    value: mlops
+    value: mlops-studio
   - name: vpc_id
     valueFrom:
       moduleMetadata:

manifests/storage-modules.yaml

@@ -0,0 +1,15 @@
+name: ecr-mlflow
+path: git::https://github.com/awslabs/idf-modules.git//modules/storage/ecr?ref=release/1.3.0&depth=1
+parameters:
+  - name: repository_name
+    value: "ecr-mlflow"
+---
+name: buckets
+path: git::https://github.com/awslabs/idf-modules.git//modules/storage/buckets?ref=release/1.3.0&depth=1
+targetAccount: primary
+targetRegion: us-east-1
+parameters:
+  - name: encryption-type
+    value: SSE
+  - name: retention-type
+    value: RETAIN

modules/mlflow/mlflow-fargate/README.md

@@ -0,0 +1,83 @@
+# Mlflow on Fargate module
+
+## Description
+
+This module runs Mlflow on AWS Fargate.
+
+By default, uses EFS for backend storage.
+
+### Architecture
+
+![Mlflow on AWS Fargate Module Architecture](docs/_static/mlflow-fargate-module-architecture.png "Mlflow on AWS Fargate Module Architecture")
+
+## Inputs/Outputs
+
+### Input Parameters
+
+#### Required
+
+- `vpc-id`: The VPC-ID that the ECS cluster will be created in.
+- `subnet-ids`: The subnets that the Fargate task will use.
+- `ecr-repository-name`: The name of the ECR repository to pull the image from.
+- `artifacts-bucket-name`: Name of the artifacts store bucket
+
+#### Optional
+
+- `ecs-cluster-name`: Name of the ECS cluster.
+- `service-name`: Name of the service.
+- `task-cpu-units`: The number of cpu units used by the Fargate task.
+- `task-memory-limit-mb`: The amount (in MiB) of memory used by the Fargate task.
+- `lb-access-logs-bucket-name`: Name of the bucket to store load balancer access logs
+- `lb-access-logs-bucket-prefix`: Prefix for load balancer access logs
+
+### Sample manifest declaration
+
+```yaml
+name: mlflow-fargate
+path: modules/mlflow/mlflow-fargate
+parameters:
+  - name: vpc-id
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: VpcId
+  - name: subnet-ids
+    valueFrom:
+      moduleMetadata:
+        group: networking
+        name: networking
+        key: PrivateSubnetIds
+  - name: ecr-repository-name
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: ecr-mlflow
+        key: EcrRepositoryName
+  - name: artifacts-bucket-name
+    valueFrom:
+      moduleMetadata:
+        group: storage
+        name: buckets
+        key: ArtifactsBucketName
+```
+
+### Module Metadata Outputs
+
+- `ECSClusterName`: Name of the ECS cluster.
+- `ServiceName`: Name of the service.
+- `LoadBalancerDNSName`: Load balancer DNS name.
+- `LoadBalancerAccessLogsBucketArn`: Load balancer access logs bucket arn
+- `EFSFileSystemId`: EFS file system id.
+
+#### Output Example
+
+```
+{
+  "ECSClusterName": "mlops-mlops-mlflow-mlflow-fargate-EcsCluster97242B84-xxxxxxxxxxxx",
+  "ServiceName": "mlops-mlops-mlflow-mlflow-fargate-MlflowLBServiceEBACC043-xxxxxxxxxxxx",
+  "LoadBalancerDNSName": "xxxxxxxxxxxx.elb.us-east-1.amazonaws.com",
+  "LoadBalancerAccessLogsBucketArn": "arn:aws:s3:::xxxxxxxxxxxx",
+  "EFSFileSystemId": "fs-xxxxxxxxxxx",
+}
+```

modules/mlflow/mlflow-fargate/app.py

@@ -0,0 +1,93 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import json
+import os
+
+import aws_cdk
+
+from stack import MlflowFargateStack
+
+
+def _param(name: str) -> str:
+    return f"SEEDFARMER_PARAMETER_{name}"
+
+
+project_name = os.getenv("SEEDFARMER_PROJECT_NAME", "")
+deployment_name = os.getenv("SEEDFARMER_DEPLOYMENT_NAME", "")
+module_name = os.getenv("SEEDFARMER_MODULE_NAME", "")
+app_prefix = f"{project_name}-{deployment_name}-{module_name}"
+
+DEFAULT_ECS_CLUSTER_NAME = None
+DEFAULT_SERVICE_NAME = None
+DEFAULT_TASK_CPU_UNITS = 4 * 1024
+DEFAULT_TASK_MEMORY_LIMIT_MB = 8 * 1024
+DEFAULT_AUTOSCALE_MAX_CAPACITY = 2
+DEFAULT_LB_ACCESS_LOGS_BUCKET_NAME = None
+DEFAULT_LB_ACCESS_LOGS_BUCKET_PREFIX = None
+
+environment = aws_cdk.Environment(
+    account=os.environ["CDK_DEFAULT_ACCOUNT"],
+    region=os.environ["CDK_DEFAULT_REGION"],
+)
+
+vpc_id = os.getenv(_param("VPC_ID"))
+subnet_ids = json.loads(os.getenv(_param("SUBNET_IDS"), "[]"))
+ecs_cluster_name = os.getenv(_param("ECS_CLUSTER_NAME"), DEFAULT_ECS_CLUSTER_NAME)
+service_name = os.getenv(_param("SERVICE_NAME"), DEFAULT_SERVICE_NAME)
+ecr_repo_name = os.getenv(_param("ECR_REPOSITORY_NAME"))
+task_cpu_units = os.getenv(_param("TASK_CPU_UNITS"), DEFAULT_TASK_CPU_UNITS)
+task_memory_limit_mb = os.getenv(_param("TASK_MEMORY_LIMIT_MB"), DEFAULT_TASK_MEMORY_LIMIT_MB)
+autoscale_max_capacity = os.getenv(_param("AUTOSCALE_MAX_CAPACITY"), DEFAULT_AUTOSCALE_MAX_CAPACITY)
+artifacts_bucket_name = os.getenv(_param("ARTIFACTS_BUCKET_NAME"))
+lb_access_logs_bucket_name = os.getenv(_param("LB_ACCESS_LOGS_BUCKET_NAME"), DEFAULT_LB_ACCESS_LOGS_BUCKET_NAME)
+lb_access_logs_bucket_prefix = os.getenv(_param("LB_ACCESS_LOGS_BUCKET_PREFIX"), DEFAULT_LB_ACCESS_LOGS_BUCKET_PREFIX)
+
+if not vpc_id:
+    raise ValueError("Missing input parameter vpc-id")
+
+if not ecr_repo_name:
+    raise ValueError("Missing input parameter ecr-repository-name")
+
+if not artifacts_bucket_name:
+    raise ValueError("Missing input parameter artifacts-bucket-name")
+
+
+app = aws_cdk.App()
+stack = MlflowFargateStack(
+    scope=app,
+    id=app_prefix,
+    app_prefix=app_prefix,
+    vpc_id=vpc_id,
+    subnet_ids=subnet_ids,
+    ecs_cluster_name=ecs_cluster_name,
+    service_name=service_name,
+    ecr_repo_name=ecr_repo_name,
+    task_cpu_units=int(task_cpu_units),
+    task_memory_limit_mb=int(task_memory_limit_mb),
+    autoscale_max_capacity=int(autoscale_max_capacity),
+    artifacts_bucket_name=artifacts_bucket_name,
+    lb_access_logs_bucket_name=lb_access_logs_bucket_name,
+    lb_access_logs_bucket_prefix=lb_access_logs_bucket_prefix,
+    env=aws_cdk.Environment(
+        account=os.environ["CDK_DEFAULT_ACCOUNT"],
+        region=os.environ["CDK_DEFAULT_REGION"],
+    ),
+)
+
+
+aws_cdk.CfnOutput(
+    scope=stack,
+    id="metadata",
+    value=stack.to_json_string(
+        {
+            "ECSClusterName": stack.cluster.cluster_name,
+            "ServiceName": stack.service.service.service_name,
+            "LoadBalancerDNSName": stack.service.load_balancer.load_balancer_dns_name,
+            "LoadBalancerAccessLogsBucketArn": stack.lb_access_logs_bucket.bucket_arn,
+            "EFSFileSystemId": stack.fs.file_system_id,
+        }
+    ),
+)
+
+app.synth()

modules/mlflow/mlflow-fargate/coverage.ini

@@ -0,0 +1,3 @@
+[run]
+omit =
+    tests/*

modules/mlflow/mlflow-fargate/deployspec.yaml

@@ -0,0 +1,35 @@
+publishGenericEnvVariables: true
+deploy:
+  phases:
+    install:
+      commands:
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    pre_build:
+      commands:
+      - echo "Prebuild stage"
+    build:
+      commands:
+      - cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports.json
+      # Export metadata
+      - seedfarmer metadata convert -f cdk-exports.json || true
+    post_build:
+      commands:
+      - echo "Build successful"
+destroy:
+  phases:
+    install:
+      commands:
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    pre_build:
+      commands:
+      - echo "Prebuild stage"
+    build:
+      commands:
+      - cdk destroy --force --app "python app.py"
+    post_build:
+      commands:
+      - echo "Destroy successful"
+
+build_type: BUILD_GENERAL1_SMALL

modules/mlflow/mlflow-fargate/docs/_static/mlflow-fargate-module-architecture.xml

@@ -0,0 +1 @@
+<mxfile modified="2024-02-16T16:21:46.131Z" host="design-inspector.a2z.com" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" etag="8RrXlRqHpwgGKitaxPPN" version="10.1.8" type="device"><diagram id="MkE2kDHcsj_W3wBg-wOKL" name="Page-1">7V1bd5s4EP41eVSOBOL26NhJN7vtbk/Spu1TjgBh02DwARzb+fU74mIDxsRp7BgS2pzEGoSuo5lPnwZ8Jg+ny08hm02+BDb3ziRsL8/k0ZkkGSqF30KwSgWqkgnGoWunIrIR3LpPPBPiTDp3bR6VMsZB4MXurCy0At/nVlySsTAMFuVsTuCVa52xMd8S3FrM25b+cO14kkp1SdvI/+LueJLXTFQjvTJleeasJ9GE2cGiIJIvz+RhGARx+mm6HHJPjF0+Ltl9Pl/G4sq1fce8edYsbVPCDY+CeWjxEY+s0J3FQQg3hZkwzX1/Jg/yVvDQZZ77xGI38NEjDyP4m+Z6zLKwbMjCmoKzOm/5lPmxa41YzIaBHzPX5+E+pad3x6Hrjz+7MQ+Zl85dzP241OtZGMx4GGdaM4ljMd+DM+kKfiB74AXj1XnErXnoxqtzNmVPgX9u80e47ARz305aAAnbZeOQTdGjG83XLQM5kyRFMRQVWbLtIGpJFjKpbkASa46qM9My0im+Stt8fXO9NawvahVosTv2ketHM9BTMZZXVjCdBT70PIKETpmOTcVBikolRBnRkKHoCuKmY5uq4lCTW4cdmmgVxXyKpmLJwnyABFNFNhRDQ1TSZUQdVUE6NjBiXNUcw5a5QY3ioMCHek3Ir9boZn4pU/d61V+vurCsFy9ReNIRhSe9wn84hQ/M38JZSdhjJvjLpF+ZuoJiWK73bTXLxnzMoXzXQuuxS/PlK8WTb0gw+jL8m+PlN/TzaWSZt08oLa28tqJ4lTu1UIwKFyXgM/liMQHFvJ2xZJYX4MVBNomnolkEPjqu5w1hjKEnIx9aACKbRZPkdnEdlB6Gg3kDD6YbZGYQx8EULqQNFZf5cueyzkSh7Xyfgd6mrSq48CjisRiMu+LSkgu9+8SDKY/DFWTOa1GzsVyVocRi47+pqqeyScF3U1KxB+N10RsN2NP4NE1LO2zS0dc3dRRiGMRBjmypsL4prG8TY4SJrKiYmYbk0F3re92GxWJxvpDPg1CsVijOEItWzIYkIdAZFK2g30vkQ4VyLNbMIS1YyXiMBbxFURzOrXgectEOk3LOLB1hyhVEOdguXTXArNqK6pgG1UzNbHcH36OJbvS1Xx4upYfRX1b0+/6HvLgYo8V//3x+G/+LFctSZGIhGwwNooYlQz9soS2ypmmWQiXV2bsfzealbVDiUF5TND91nA1OVPVi4aFc6I06jhM3lIrMXHD3dZjLoESzmi/peOHuWq/8OLPKfvje1snMlH8N9PnNz8d/RnEUDu+RojT64VngioUGn5UL+AHjPMRnClwZitS5pFQE1bRWFpDtlCijLKimtbKAVIsnlfpJtYEFwVaqVDyu1I8LDYQf+SKYx7Do+XC9lRfoRIACFxS/AkEKAKUOvzigYxmZQKQ8nQ28KBU248lUTpeJXT9ni0jC52MARrOkzmsrcZk1l+/F1EMBcRg88LxRYJwlquuE1uOlKkSKA1ELy1Ied4SmRdADseKT1EjGWasLVQwGF9qFXsRf+JQwyyBlmKVu4yw5h1RFnAVjdTCc1bTs2oGzGu14c/PbZsd7vFGDN3oc3Xkc3bhEE7TwksoljLGoHKzjVVKzNQEDAtWmMGXfer/7D36w8L8KfrqVAPVNUV0zNTJhgBi8fQCZqjYCMm6PeQ4TgjCeBOPAZ97lRnpRpk4KKIQv3finEAPCSVO/skzi82hZTKwKia/QBfB+YtxSmQ+e8GdeqEj82pQqkpuiktSqmKoW9pvH8SrDQWweBwJkrbv1ORAwJKknHTrR++cQRNEI1O9C8lMaFo5543aFaAcEJSH3QJEfeaWEIwGMVI06CzDy5vcAowMAo9te4tjgQVWIoqkSRho1YUg1zKEpho1Mw3QU1VQtkx4XPFx4zH/4F6b3evQCJvjAhz7b42ITx8KqypGsq0LTOUa6jClSLe5QQIyYKfgA3dCO3Q+VaCYHz4QsBxYrZZggpik20gm3ZcDCkpLtJtvJNV0Ob+GmoTePhFN+LevkgE9lMS8DnQYFexnxVEs+1RFQtSTUNhFVypZQQzU1VIV1Mm1bSLaz5WzStrBOVkedVe8mNXeTyt27iasKeQP/r8R8bBFacO1K0y8xLVwbuSEUlHp7XyC2Cq8E94ywMoTlV8NEOcm/Kk2Uc1CfhQJ/DSI3K359XNdMUlnQKrGKSieDz5BsLPHCkHDcpWjHDtYtNzYp53YByVr2LVf+0zFecoZWcsZL0tQtxkuvIbz0Y58rdiXU4U18YWfAqIU1QH+cIolpAARVxhEzFQUxaKSjU1OyjAODuhawXZ3mgire/ASsUIcOMFuHxG5ejb+4Fe6DvaQee/XY611hL6H4p8NdqoZLuKvupPEkuCsnpruJu1oaLXJ03AUKZRiORBCTCEYUawb4MIkh1bEZAC/i8N0Haj3uOg3uuunRVuvQ1hpD3YL5xgPwYA5L8t5C4bwebNXCqkjeB1VpParqDqpS8UCWtZehKknTCFE/EKoCvW8PqJL0lpBZWqdB1YEPRDoDqjin1JQNhixqmgCqwOWbhkqQ5FCdW8SQea44PahqB6ja4bV7mNUamFULlhqDnuptktGInF4T85THKq0DkDoWq5Q9BbZHrNIBHfCxQpUaJr+z7tT4oO703QUqdcjG9yFLTbay8yFLx+5GByKWNodiHougXsj1OWA2/IHdMfOtXaFK9YdiaRn3HpRwbyb3J/bjeVyi94xOdxid/pxsn3Oy+rVwOpYH/p0rZZ6nLUFLeqeB6YG9YWeAqa0bkuMwDak6xojqWEO6YRCkKAqWiGzz9eFse4Dpx+Z5Gnx8z/V0mOvZ8ezw0R5wO8mzZNo2QbOj31IHGJqmKeuAJ2xu/odzhe+Oo+kfJusmM3P8p7AOTc002/APzc00xTBf5c9/4W8senh1OHMEbjWez+7jpLDn+RrSHILz7O69ACb23MiXeYPsbToVkgHji8srqW7n/gSTfl7qpBjeDWsBU5ewFqfanFMiJXROYXdONHUtKmzQlZoNOtXynf3xnivqdizGoe1ij0zavEnv36PTahqi6rt67uH9cQ/N+OBjBJo0Q9vnI006zGN0BS00N79HCx1ACz2P0Wke41T7/7cJMfnYNMbB0ERzeEiPJvZBEyd5xV5y6yAM2aqQIQvn2ZT8VQg2ZIiacxx5nAIurcRn88uSWlH5tAUbPV535QBQpyvBC83N76FOD3V6qNNDnT+HOv37/5pPbFKuC18PvsDvm8Db8Vj0Cw5twqSQPeCT1AifXhUOWn8QM9SITK52ndq0LtAzASQ8vHzkaZgx2R38GSR48uxEx0SKfPLXnzQbsa6ioAPb4B4F9cdD/fHQHx4P1TjKE5wQtR5Od5PJocdick5Cvsj7ki/6KciXw3h22m3PfuD3d/Wevec3en7jRfyG/D74jaM/9NxxfuN7dICvNWBW5i2eBRJF57gNJPLd+3zqDZIi9yAakv5eMOthnICMukdU67mIAgqp5VJOxRYQUjlHqXmvl1zDFqhHZwvkbmOKA1u01mCKZ9w50y0F27aNGDN0RHWLI0aYhRRmKA7smSnXDjwyLSALOr2VTm1yv3vu3O75bnJ39/D34NMP5XYJeOW/H4M7+i9Sj/f6rtqvLMyv/MFXFrY/nmLHENc8Ztrh7yxs0qMO+N/m5rfN/z63r2jWuK7vj9r3BXDvfkPek0jvjkRq2ab+8kp8zdFj4M2nr49X4E50H8UMBi2094I7/XveO/RWsP497/u8Fay4BE759YVq5V1g8psHkjSjoa5C0/7Joe4ghT6Q5H2zX2Xw0rNgb4Y2xagFQVzI+kmozhexVEH4Pw==</diagram></mxfile>

modules/mlflow/mlflow-fargate/pyproject.toml

@@ -0,0 +1,35 @@
+[tool.black]
+line-length = 120
+target-version = ["py36", "py37", "py38"]
+exclude = '''
+/(
+    \.eggs
+  | \.git
+  | \.hg
+  | \.mypy_cache
+  | \.tox
+  | \.venv
+  | \.env
+  | _build
+  | buck-out
+  | build
+  | dist
+  | codeseeder.out
+)/
+'''
+
+[tool.isort]
+multi_line_output = 3
+include_trailing_comma = true
+force_grid_wrap = 0
+use_parentheses = true
+ensure_newline_before_comments = true
+line_length = 120
+py_version = 36
+skip_gitignore = false
+
+[tool.pytest.ini_options]
+addopts = "-v --cov=. --cov-report term --cov-config=coverage.ini --cov-fail-under=80"
+pythonpath = [
+  "."
+]

modules/mlflow/mlflow-fargate/requirements-dev.in

@@ -0,0 +1,14 @@
+awscli
+black
+cdk-nag
+cfn-lint
+check-manifest
+flake8
+isort
+mypy
+pip-tools
+pydot
+pyroma
+pytest
+types-PyYAML
+types-setuptools