Added a new template to import huggingface models by taking access to… (

#71) * Added a new template to import huggingface models by taking access token and model id as parameters * Added a new template to import huggingface models by taking access token and model id as parameters * Added a new template to import huggingface models by taking access token and model id as parameters * Fixed formatting by running the shell script fix.sh * Fixed typing errors and changed hugging face access token to be read from a Secrets Manager Secret * Fixed typing errors * Fixed formating * Fixed formating and typing errors based on python 3.9 * Fixed formating and typing errors based on python 3.9
awslabs · May 3, 2024 · b474d12 · b474d12
1 parent 0f5a208
commit b474d12
Show file tree

Hide file tree

Showing 22 changed files with 1,236 additions and 8 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - added EFS removal policy to `mlflow-fargate` module
 - added `mwaa` module with example dag which demonstrates the MLOps in Airflow
 - added `sagemaker-hugging-face-endpoint` module
+- added `hf_import_models` template to import hugging face models
 
 ### **Changed**
 

diff --git a/README.md b/README.md
@@ -23,13 +23,13 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).
 
 ### SageMaker Modules
 
-| Type                                                                                                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
-|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)                                                   | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config                                                                                                                                                                                                                                                                                                                                                                                                                                          |
-| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md)                                               | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
-| [SageMaker Project Templates via Service Catalog Module](modules/sagemaker/sagemaker-templates-service-catalog/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog. Available templates:<br/> - [Train a model on Abalone dataset using XGBoost](modules/sagemaker/sagemaker-templates-service-catalog/README.md#train-a-model-on-abalone-dataset-with-xgboost-template)<br/>- [Perform batch inference](modules/sagemaker/sagemaker-templates-service-catalog/README.md#batch-inference-template)<br/>- [Multi-account model deployment](modules/sagemaker/sagemaker-templates-service-catalog/README.md#multi-account-model-deployment-template) |
-| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md)                                      | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
-| [SageMaker Custom Kernel Module](modules/sagemaker/sagemaker-custom-kernel/README.md)                                     | Builds custom kernel for SageMaker Studio from a Dockerfile                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| Type                                                                                                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)                                                   | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md)                                               | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| [SageMaker Project Templates via Service Catalog Module](modules/sagemaker/sagemaker-templates-service-catalog/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog. Available templates:<br/> - [Train a model on Abalone dataset using XGBoost](modules/sagemaker/sagemaker-templates-service-catalog/README.md#train-a-model-on-abalone-dataset-with-xgboost-template)<br/>- [Perform batch inference](modules/sagemaker/sagemaker-templates-service-catalog/README.md#batch-inference-template)<br/>- [Multi-account model deployment](modules/sagemaker/sagemaker-templates-service-catalog/README.md#multi-account-model-deployment-template) <br/>- [HuggingFace model import template](modules/sagemaker/sagemaker-templates-service-catalog/README.md#huggingface-model-import-template) |
+| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md)                                      | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| [SageMaker Custom Kernel Module](modules/sagemaker/sagemaker-custom-kernel/README.md)                                     | Builds custom kernel for SageMaker Studio from a Dockerfile                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 
 
 ### Mlflow Modules

diff --git a/modules/sagemaker/sagemaker-templates-service-catalog/README.md b/modules/sagemaker/sagemaker-templates-service-catalog/README.md
@@ -26,6 +26,14 @@ This project template contains SageMaker pipeline that performs batch inference.
 
 ![Batch Inference Template](docs/_static/batch-inference-template.png "Batch Inference Template Architecture")
 
+#### Huggingface Model Import Template
+
+This project template contains SageMaker pipeline that imports a hugging face model based on model id and access 
+token inputs.
+
+![Huggingface model import template](docs/_static/huggingface-model-import.png "Hugging Face Model Import Template 
+Architecture")
+
 #### Multi-account Model Deployment Template
 
 The template contains an example CI/CD pipeline to deploy the model endpoints to multiple AWS accounts. 

diff --git a/...r/sagemaker-templates-service-catalog/docs/_static/huggingface-model-import.png b/...r/sagemaker-templates-service-catalog/docs/_static/huggingface-model-import.png
diff --git a/...s/sagemaker/sagemaker-templates-service-catalog/docs/_static/huggingface-model-import.xml b/...s/sagemaker/sagemaker-templates-service-catalog/docs/_static/huggingface-model-import.xml