[Fix] Model uploader's jekins trigger parameter fix (#402)

* [Feature] Add a workflow parameter that model uploader can specific a customize prefix. Signed-off-by: conggguan <[email protected]> * [Fix] To fix the Jekins trigger's wrong folder parameters bug. Signed-off-by: conggguan <[email protected]> * [Comments] Add a comments for model uploader workflow's [Initiate folders]. Signed-off-by: conggguan <[email protected]> --------- Signed-off-by: conggguan <[email protected]>
opensearch-project · Aug 8, 2024 · d1d968d · d1d968d
1 parent 026ac2a
commit d1d968d
Show file tree

Hide file tree

Showing 4 changed files with 14 additions and 15 deletions.
diff --git a/.github/workflows/model_uploader.yml b/.github/workflows/model_uploader.yml
@@ -74,15 +74,25 @@ jobs:
          echo "This workflow should only be triggered on 'main' branch"
          exit 1
     - name: Initiate folders
+      # This scripts init the folders path variables.
+      # 1. Retrieves the input model_id.
+      # 2. If upload_prefix is provided, constructs model_prefix using upload_prefix and model_source.
+      #    - model_prefix: "ml-models/{model_source}/{upload_prefix}"
+      # 3. If upload_prefix is not provided, it constructs model_prefix using model_source and the prefix part of model_id.
+      #    - The prefix part is the substring before the first '/' in model_id.
+      #    Example:
+      #    - Given model_id: "opensearch-project/opensearch-neural-sparse-encoding-v1"
+      #    - model_prefix: "ml-models/{model_source}/opensearch-project"
+      # 4. Constructs model_folder and model_prefix_folder.
       id: init_folders
       run: |
         model_id=${{ github.event.inputs.model_id }}
-        echo "model_folder=ml-models/${{github.event.inputs.model_source}}/${model_id}" >> $GITHUB_OUTPUT
         if [[ -n "${{ github.event.inputs.upload_prefix }}" ]]; then
           model_prefix="ml-models/${{ github.event.inputs.model_source }}/${{ github.event.inputs.upload_prefix }}"
         else
           model_prefix="ml-models/${{ github.event.inputs.model_source }}/${model_id%%/*}"
         fi
+        echo "model_folder=$model_prefix/${model_id##*/}" >> $GITHUB_OUTPUT
         echo "model_prefix_folder=$model_prefix" >> $GITHUB_OUTPUT
     - name: Initiate workflow_info
       id: init_workflow_info
@@ -446,4 +456,4 @@ jobs:
           version=${{ github.event.inputs.model_version }}
           format=${{ github.event.inputs.tracing_format }}
           jenkins_params="{\"BASE_DOWNLOAD_PATH\":\"$base_download_path\", \"VERSION\":\"$version\", \"FORMAT\":\"$format\"}"
-          sh utils/model_uploader/trigger_ml_models_release.sh $jenkins_trigger_token "$jenkins_params"
+          sh utils/model_uploader/trigger_ml_models_release.sh $jenkins_trigger_token "$jenkins_params"
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -40,9 +40,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Removed pandas version pin in nox tests by @rawwar ([#368](https://github.com/opensearch-project/opensearch-py-ml/pull/368))
 - Switch AL2 to AL2023 agent and DockerHub to ECR images in ml-models.JenkinsFile ([#377](https://github.com/opensearch-project/opensearch-py-ml/pull/377))
 - Refactored validators in ML Commons' client([#385](https://github.com/opensearch-project/opensearch-py-ml/pull/385))
-- Update model upload history -  opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill (v.1.0.0)(TORCH_SCRIPT) by @dhrubo-os ([#400](https://github.com/opensearch-project/opensearch-py-ml/pull/400))
 
 ### Fixed
+- Fix the wrong input parameter for model_uploader's base_download_path in jekins trigger.([#402](https://github.com/opensearch-project/opensearch-py-ml/pull/402))
 - Enable make_model_config_json to add model description to model config file by @thanawan-atc in ([#203](https://github.com/opensearch-project/opensearch-py-ml/pull/203))
 - Correct demo_ml_commons_integration.ipynb by @thanawan-atc in ([#208](https://github.com/opensearch-project/opensearch-py-ml/pull/208))
 - Handle the case when the model max length is undefined in tokenizer by @thanawan-atc in ([#219](https://github.com/opensearch-project/opensearch-py-ml/pull/219))

diff --git a/utils/model_uploader/upload_history/MODEL_UPLOAD_HISTORY.md b/utils/model_uploader/upload_history/MODEL_UPLOAD_HISTORY.md
@@ -20,5 +20,4 @@ The following table shows sentence transformer model upload history.
 |2023-08-31 15:22:19|@dhrubo-os|`sentence-transformers/msmarco-distilbert-base-tas-b`|1.0.2|TORCH_SCRIPT|N/A|N/A|6042401385|
 |2023-09-13 18:03:32|@dhrubo-os|`sentence-transformers/distiluse-base-multilingual-cased-v1`|1.0.1|TORCH_SCRIPT|N/A|N/A|6178024517|
 |2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|ONNX|N/A|N/A|6568285400|
-|2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|TORCH_SCRIPT|N/A|N/A|6568285400|
-|2024-08-06 12:42:00|@dhrubo-os|`opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill`|1.0.0|TORCH_SCRIPT|N/A|N/A|10271804648|
+|2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|TORCH_SCRIPT|N/A|N/A|6568285400|
diff --git a/utils/model_uploader/upload_history/supported_models.json b/utils/model_uploader/upload_history/supported_models.json
@@ -48,15 +48,5 @@
         "Embedding Dimension": "N/A",
         "Pooling Mode": "N/A",
         "Workflow Run ID": "6568285400"
-    },
-    {
-        "Model Uploader": "@dhrubo-os",
-        "Upload Time": "2024-08-06 12:42:00",
-        "Model ID": "opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill",
-        "Model Version": "1.0.0",
-        "Model Format": "TORCH_SCRIPT",
-        "Embedding Dimension": "N/A",
-        "Pooling Mode": "N/A",
-        "Workflow Run ID": "10271804648"
     }
 ]