Huggingface Model Deployer (zenml-io#2376)

* Initial implementation of huggingface model deployer * Add missing step init * Simplify modify_endpoint_name function and fix docstrings * Formatting logger * Add License to new files * Enhancements as per PR review comments * Add logging message to catch KeyError * Remove duplicate variable * Reorder lines for clarity * Add docs for huggingface model deployer * Fix CI errors * Fix get_model_info function arguments * More CI fixes * Add minimal supported version for Inference Endpoint API in huggingface_hub * Relax 'adlfs' package requirement in azure integrations * update TOC (zenml-io#2406) * Relax 's3fs' version in s3 integration * Bugs fixed running a test deployment pipeline * Add deployment pipelines to huggingface integration test * Remove not required check on service running in tests * Address PR comments on documentation and suggested renaming in code * Add partial test for huggingface_deployment * Fix typo in test function * Update pyproject.toml This should allow the dependencies to resolve. * Update pyproject.toml * Relax gcfs * Update model deployers table * Fix lint issue --------- Co-authored-by: Andrei Vishniakov <[email protected]> Co-authored-by: Safoine El Khabich <[email protected]> Co-authored-by: Alex Strick van Linschoten <[email protected]>
adtygan · Mar 20, 2024 · a237a53 · a237a53
1 parent d19d0fb
commit a237a53
Show file tree

Hide file tree

Showing 24 changed files with 1,486 additions and 7 deletions.
diff --git a/docs/book/stacks-and-components/component-guide/model-deployers/huggingface.md b/docs/book/stacks-and-components/component-guide/model-deployers/huggingface.md
@@ -0,0 +1,154 @@
+---
+description: Deploying models to Huggingface Inference Endpoints with Hugging Face :hugging_face:.
+---
+
+# Hugging Face :hugging_face:
+
+Hugging Face Inference Endpoints provides a secure production solution to easily deploy any `transformers`, `sentence-transformers`, and `diffusers` models on a dedicated and autoscaling infrastructure managed by Hugging Face. An Inference Endpoint is built from a model from the [Hub](https://huggingface.co/models).
+
+This service provides dedicated and autoscaling infrastructure managed by Hugging Face, allowing you to deploy models without dealing with containers and GPUs.
+
+## When to use it?
+
+You should use Hugging Face Model Deployer:
+
+* if you want to deploy [Transformers, Sentence-Transformers, or Diffusion models](https://huggingface.co/docs/inference-endpoints/supported_tasks) on dedicated and secure infrastructure.
+* if you prefer a fully-managed production solution for inference without the need to handle containers and GPUs.
+* if your goal is to turn your models into production-ready APIs with minimal infrastructure or MLOps involvement
+* Cost-effectiveness is crucial, and you want to pay only for the raw compute resources you use.
+* Enterprise security is a priority, and you need to deploy models into secure offline endpoints accessible only via a direct connection to your Virtual Private Cloud (VPCs).
+
+If you are looking for a more easy way to deploy your models locally, you can use the [MLflow Model Deployer](mlflow.md) flavor.
+
+## How to deploy it?
+
+The Hugging Face Model Deployer flavor is provided by the Hugging Face ZenML integration, so you need to install it on your local machine to be able to deploy your models. You can do this by running the following command:
+
+```bash
+zenml integration install huggingface -y
+```
+
+To register the Hugging Face model deployer with ZenML you need to run the following command:
+
+```bash
+zenml model-deployer register <MODEL_DEPLOYER_NAME> --flavor=huggingface --token=<YOUR_HF_TOKEN> --namespace=<YOUR_HF_NAMESPACE>
+```
+
+Here,
+
+* `token` parameter is the Hugging Face authentication token. It can be managed through [Hugging Face settings](https://huggingface.co/settings/tokens).
+* `namespace` parameter is used for listing and creating the inference endpoints. It can take any of the following values, username or organization name or `*` depending on where the inference endpoint should be created.
+
+We can now use the model deployer in our stack.
+
+```bash
+zenml stack update <CUSTOM_STACK_NAME> --model-deployer=<MODEL_DEPLOYER_NAME>
+```
+
+See the [huggingface_model_deployer_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-seldon/#zenml.integrations.huggingface.steps.huggingface_deployer.huggingface_model_deployer_step) for an example of using the Hugging Face Model Deployer to deploy a model inside a ZenML pipeline step.
+
+## Configuration
+
+Within the `HuggingFaceServiceConfig` you can configure:
+
+* `model_name`: the name of the model in ZenML.
+* `endpoint_name`: the name of the inference endpoint. We add a prefix `zenml-` and first 8 characters of the service uuid as a suffix to the endpoint name.
+* `repository`: The repository name in the user’s namespace (`{username}/{model_id}`) or in the organization namespace (`{organization}/{model_id}`) from the Hugging Face hub.
+* `framework`: The machine learning framework used for the model (e.g. `"custom"`, `"pytorch"` )
+* `accelerator`: The hardware accelerator to be used for inference. (e.g. `"cpu"`, `"gpu"`)
+* `instance_size`: The size of the instance to be used for hosting the model (e.g. `"large"`, `"xxlarge"`)
+* `instance_type`: Inference Endpoints offers a selection of curated CPU and GPU instances. (e.g. `"c6i"`, `"g5.12xlarge"`)
+* `region`: The cloud region in which the Inference Endpoint will be created (e.g. `"us-east-1"`, `"eu-west-1"` for `vendor = aws` and `"eastus"` for Microsoft Azure vendor.).
+* `vendor`: The cloud provider or vendor where the Inference Endpoint will be hosted (e.g. `"aws"`).
+* `token`: The Hugging Face authentication token. It can be managed through [huggingface settings](https://huggingface.co/settings/tokens). The same token can be passed used while registering the Hugging Face model deployer.
+* `account_id`: (Optional) The account ID used to link a VPC to a private Inference Endpoint (if applicable).
+* `min_replica`: (Optional) The minimum number of replicas (instances) to keep running for the Inference Endpoint. Defaults to `0`.
+* `max_replica`: (Optional) The maximum number of replicas (instances) to scale to for the Inference Endpoint. Defaults to `1`.
+* `revision`: (Optional) The specific model revision to deploy on the Inference Endpoint for the Hugging Face repository .
+* `task`: Select a supported [Machine Learning Task](https://huggingface.co/docs/inference-endpoints/supported_tasks). (e.g. `"text-classification"`, `"text-generation"`)
+* `custom_image`: (Optional) A custom Docker image to use for the Inference Endpoint.
+* `namespace`: The namespace where the Inference Endpoint will be created. The same namespace can be passed used while registering the Hugging Face model deployer.
+* `endpoint_type`: (Optional) The type of the Inference Endpoint, which can be `"protected"`, `"public"` (default) or `"private"`.
+
+For more information and a full list of configurable attributes of the Hugging Face Model Deployer, check out
+the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface/#zenml.integrations.huggingface.model_deployers) and Hugging Face endpoint [code](https://github.com/huggingface/huggingface_hub/blob/5e3b603ccc7cd6523d998e75f82848215abf9415/src/huggingface_hub/hf_api.py#L6957).
+
+### Run inference on a provisioned inference endpoint
+
+The following code example shows how to run inference against a provisioned inference endpoint:
+
+```python
+from typing import Annotated
+from zenml import step, pipeline
+from zenml.integrations.huggingface.model_deployers import HuggingFaceModelDeployer
+from zenml.integrations.huggingface.services import HuggingFaceDeploymentService
+
+
+# Load a prediction service deployed in another pipeline
+@step(enable_cache=False)
+def prediction_service_loader(
+    pipeline_name: str,
+    pipeline_step_name: str,
+    running: bool = True,
+    model_name: str = "default",
+) -> HuggingFaceDeploymentService:
+    """Get the prediction service started by the deployment pipeline.
+
+    Args:
+        pipeline_name: name of the pipeline that deployed the MLflow prediction
+            server
+        step_name: the name of the step that deployed the MLflow prediction
+            server
+        running: when this flag is set, the step only returns a running service
+        model_name: the name of the model that is deployed
+    """
+    # get the Hugging Face model deployer stack component
+    model_deployer = HuggingFaceModelDeployer.get_active_model_deployer()
+
+    # fetch existing services with same pipeline name, step name and model name
+    existing_services = model_deployer.find_model_server(
+        pipeline_name=pipeline_name,
+        pipeline_step_name=pipeline_step_name,
+        model_name=model_name,
+        running=running,
+    )
+
+    if not existing_services:
+        raise RuntimeError(
+            f"No Hugging Face inference endpoint deployed by step "
+            f"'{pipeline_step_name}' in pipeline '{pipeline_name}' with name "
+            f"'{model_name}' is currently running."
+        )
+
+    return existing_services[0]
+
+
+# Use the service for inference
+@step
+def predictor(
+    service: HuggingFaceDeploymentService,
+    data: str
+) -> Annotated[str, "predictions"]:
+    """Run a inference request against a prediction service"""
+
+    prediction = service.predict(data)
+    return prediction
+
+
+@pipeline
+def huggingface_deployment_inference_pipeline(
+    pipeline_name: str, pipeline_step_name: str = "huggingface_model_deployer_step",
+):
+    inference_data = ...
+    model_deployment_service = prediction_service_loader(
+        pipeline_name=pipeline_name,
+        pipeline_step_name=pipeline_step_name,
+    )
+    predictions = predictor(model_deployment_service, inference_data)
+```
+
+For more information and a full list of configurable attributes of the Hugging Face Model Deployer, check out
+the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface/#zenml.integrations.huggingface.model_deployers).
+
+<!-- For scarf -->
+<figure><img alt="ZenML Scarf" referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" /></figure>
diff --git a/docs/book/stacks-and-components/component-guide/model-deployers/model-deployers.md b/docs/book/stacks-and-components/component-guide/model-deployers/model-deployers.md
@@ -44,6 +44,7 @@ integrations:
 | [MLflow](mlflow.md)                | `mlflow`  | `mlflow`      | Deploys ML Model locally                                                     |
 | [BentoML](bentoml.md)              | `bentoml` | `bentoml`     | Build and Deploy ML models locally or for production grade (Cloud, K8s)      |
 | [Seldon Core](seldon.md)           | `seldon`  | `seldon Core` | Built on top of Kubernetes to deploy models for production grade environment |
+| [Hugging Face](huggingface.md) | `huggingface` | `huggingface` | Deploys ML model on Hugging Face Inference Endpoints |
 | [Custom Implementation](custom.md) | _custom_  |               | Extend the Artifact Store abstraction and provide your own implementation    |
 
 {% hint style="info" %}
@@ -85,6 +86,7 @@ zenml model-deployer register seldon --flavor=seldon \
    ...
    zenml stack register seldon_stack -m default -a aws -o default -d seldon
    ```
+
 2. Implements the continuous deployment logic necessary to deploy models in a way that updates an existing model server
    that is already serving a previous version of the same model instead of creating a new model server for every new
    model version. Every model server that the Model Deployer provisions externally to deploy a model is represented

diff --git a/docs/book/toc.md b/docs/book/toc.md
@@ -125,6 +125,7 @@
     * [MLflow](stacks-and-components/component-guide/model-deployers/mlflow.md)
     * [Seldon](stacks-and-components/component-guide/model-deployers/seldon.md)
     * [BentoML](stacks-and-components/component-guide/model-deployers/bentoml.md)
+    * [Hugging Face](stacks-and-components/component-guide/model-deployers/huggingface.md)
     * [Develop a Custom Model Deployer](stacks-and-components/component-guide/model-deployers/custom.md)
   * [Step Operators](stacks-and-components/component-guide/step-operators/step-operators.md)
     * [Amazon SageMaker](stacks-and-components/component-guide/step-operators/sagemaker.md)

diff --git a/docs/mocked_libs.json b/docs/mocked_libs.json
@@ -106,6 +106,8 @@
     "great_expectations.types",
     "hvac",
     "hvac.exceptions",
+    "huggingface_hub",
+    "huggingface_hub.utils",
     "kfp",
     "kfp.compiler",
     "kfp.v2",

diff --git a/pyproject.toml b/pyproject.toml
@@ -448,5 +448,6 @@ module = [
     "mlstacks.*",
     "matplotlib.*",
     "IPython.*",
+    "huggingface_hub.*"
 ]
 ignore_missing_imports = true
diff --git a/src/zenml/integrations/huggingface/__init__.py b/src/zenml/integrations/huggingface/__init__.py
@@ -12,9 +12,14 @@
 #  or implied. See the License for the specific language governing
 #  permissions and limitations under the License.
 """Initialization of the Huggingface integration."""
+from typing import List, Type
 
 from zenml.integrations.constants import HUGGINGFACE
 from zenml.integrations.integration import Integration
+from zenml.stack import Flavor
+
+HUGGINGFACE_MODEL_DEPLOYER_FLAVOR = "huggingface"
+HUGGINGFACE_SERVICE_ARTIFACT = "hf_deployment_service"
 
 
 class HuggingfaceIntegration(Integration):
@@ -31,6 +36,20 @@ class HuggingfaceIntegration(Integration):
     def activate(cls) -> None:
         """Activates the integration."""
         from zenml.integrations.huggingface import materializers  # noqa
+        from zenml.integrations.huggingface import services
+
+    @classmethod
+    def flavors(cls) -> List[Type[Flavor]]:
+        """Declare the stack component flavors for the Huggingface integration.
+
+        Returns:
+            List of stack component flavors for this integration.
+        """
+        from zenml.integrations.huggingface.flavors import (
+            HuggingFaceModelDeployerFlavor,
+        )
+
+        return [HuggingFaceModelDeployerFlavor]
 
 
 HuggingfaceIntegration.check_installation()
diff --git a/src/zenml/integrations/huggingface/flavors/__init__.py b/src/zenml/integrations/huggingface/flavors/__init__.py
@@ -0,0 +1,26 @@
+#  Copyright (c) ZenML GmbH 2024. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at:
+#
+#       https://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+#  or implied. See the License for the specific language governing
+#  permissions and limitations under the License.
+"""Hugging Face integration flavors."""
+
+from zenml.integrations.huggingface.flavors.huggingface_model_deployer_flavor import (  # noqa
+    HuggingFaceModelDeployerConfig,
+    HuggingFaceModelDeployerFlavor,
+    HuggingFaceBaseConfig,
+)
+
+__all__ = [
+    "HuggingFaceModelDeployerConfig",
+    "HuggingFaceModelDeployerFlavor",
+    "HuggingFaceBaseConfig",
+]
diff --git a/src/zenml/integrations/huggingface/flavors/huggingface_model_deployer_flavor.py b/src/zenml/integrations/huggingface/flavors/huggingface_model_deployer_flavor.py
@@ -0,0 +1,131 @@
+#  Copyright (c) ZenML GmbH 2024. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at:
+#
+#       https://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+#  or implied. See the License for the specific language governing
+#  permissions and limitations under the License.
+"""Hugging Face model deployer flavor."""
+
+from typing import TYPE_CHECKING, Any, Dict, Optional, Type
+
+from pydantic import BaseModel
+
+from zenml.integrations.huggingface import HUGGINGFACE_MODEL_DEPLOYER_FLAVOR
+from zenml.model_deployers.base_model_deployer import (
+    BaseModelDeployerConfig,
+    BaseModelDeployerFlavor,
+)
+from zenml.utils.secret_utils import SecretField
+
+if TYPE_CHECKING:
+    from zenml.integrations.huggingface.model_deployers.huggingface_model_deployer import (
+        HuggingFaceModelDeployer,
+    )
+
+
+class HuggingFaceBaseConfig(BaseModel):
+    """Hugging Face Inference Endpoint configuration."""
+
+    endpoint_name: str = "zenml-"
+    repository: Optional[str] = None
+    framework: Optional[str] = None
+    accelerator: Optional[str] = None
+    instance_size: Optional[str] = None
+    instance_type: Optional[str] = None
+    region: Optional[str] = None
+    vendor: Optional[str] = None
+    token: Optional[str] = None
+    account_id: Optional[str] = None
+    min_replica: int = 0
+    max_replica: int = 1
+    revision: Optional[str] = None
+    task: Optional[str] = None
+    custom_image: Optional[Dict[str, Any]] = None
+    namespace: Optional[str] = None
+    endpoint_type: str = "public"
+
+
+class HuggingFaceModelDeployerConfig(
+    BaseModelDeployerConfig, HuggingFaceBaseConfig
+):
+    """Configuration for the Hugging Face model deployer.
+
+    Attributes:
+        token: Hugging Face token used for authentication
+        namespace: Hugging Face namespace used to list endpoints
+    """
+
+    token: str = SecretField()
+
+    # The namespace to list endpoints for. Set to `"*"` to list all endpoints
+    # from all namespaces (i.e. personal namespace and all orgs the user belongs to).
+    namespace: str
+
+
+class HuggingFaceModelDeployerFlavor(BaseModelDeployerFlavor):
+    """Hugging Face Endpoint model deployer flavor."""
+
+    @property
+    def name(self) -> str:
+        """Name of the flavor.
+
+        Returns:
+            The name of the flavor.
+        """
+        return HUGGINGFACE_MODEL_DEPLOYER_FLAVOR
+
+    @property
+    def docs_url(self) -> Optional[str]:
+        """A url to point at docs explaining this flavor.
+
+        Returns:
+            A flavor docs url.
+        """
+        return self.generate_default_docs_url()
+
+    @property
+    def sdk_docs_url(self) -> Optional[str]:
+        """A url to point at SDK docs explaining this flavor.
+
+        Returns:
+            A flavor SDK docs url.
+        """
+        return self.generate_default_sdk_docs_url()
+
+    @property
+    def logo_url(self) -> str:
+        """A url to represent the flavor in the dashboard.
+
+        Returns:
+            The flavor logo.
+        """
+        return "https://public-flavor-logos.s3.eu-central-1.amazonaws.com/model_registry/huggingface.png"
+
+    @property
+    def config_class(self) -> Type[HuggingFaceModelDeployerConfig]:
+        """Returns `HuggingFaceModelDeployerConfig` config class.
+
+        Returns:
+            The config class.
+        """
+        return HuggingFaceModelDeployerConfig
+
+    @property
+    def implementation_class(self) -> Type["HuggingFaceModelDeployer"]:
+        """Implementation class for this flavor.
+
+        Returns:
+            The implementation class.
+        """
+        from zenml.integrations.huggingface.model_deployers.huggingface_model_deployer import (
+            HuggingFaceModelDeployer,
+        )
+
+        return HuggingFaceModelDeployer