From 0746779156d7c6c3f63b12af11558034f82f9049 Mon Sep 17 00:00:00 2001 From: Rakavitha Kodhandapani Date: Tue, 31 Dec 2024 13:33:56 +0530 Subject: [PATCH 01/11] added a note about deprecation of seldon and tensorflow protocol --- doc/source/analytics/explainers.md | 3 +++ doc/source/graph/protocols.md | 2 ++ doc/source/production/optimization.md | 3 +++ doc/source/reference/upgrading.md | 4 +++- examples/models/lightgbm_custom_server/iris.ipynb | 5 ++++- notebooks/backwards_compatability.ipynb | 3 ++- notebooks/protocol_examples.ipynb | 2 ++ notebooks/server_examples.ipynb | 2 ++ 8 files changed, 21 insertions(+), 3 deletions(-) diff --git a/doc/source/analytics/explainers.md b/doc/source/analytics/explainers.md index 304828715c..d929007d68 100644 --- a/doc/source/analytics/explainers.md +++ b/doc/source/analytics/explainers.md @@ -45,6 +45,9 @@ For an e2e example, please check AnchorTabular notebook [here](../examples/iris_ ## Explain API +**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. + + For the Seldon Protocol an endpoint path will be exposed for: ``` diff --git a/doc/source/graph/protocols.md b/doc/source/graph/protocols.md index 0ab9887fd4..f237207566 100644 --- a/doc/source/graph/protocols.md +++ b/doc/source/graph/protocols.md @@ -1,5 +1,7 @@ # Protocols +**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. + Tensorflow protocol is only available in version >=1.1. Seldon Core supports the following data planes: diff --git a/doc/source/production/optimization.md b/doc/source/production/optimization.md index 92c12dbe8d..6a874193c6 100644 --- a/doc/source/production/optimization.md +++ b/doc/source/production/optimization.md @@ -9,6 +9,9 @@ Using the Seldon python wrapper there are various optimization areas one needs t ### Seldon Protocol Payload Types with REST and gRPC +**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. + + Depending on whether you want to use REST or gRPC and want to send tensor data the format of the request will have a deserialization/serialization cost in the python wrapper. This is investigated in a [python serialization notebook](../examples/python_serialization.html). The conclusions are: diff --git a/doc/source/reference/upgrading.md b/doc/source/reference/upgrading.md index bfc005c9c9..fd1ac436f0 100644 --- a/doc/source/reference/upgrading.md +++ b/doc/source/reference/upgrading.md @@ -93,10 +93,12 @@ Only the v1 versions of the CRD will be supported moving forward. The v1beta1 ve ### Model Health Checks +**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. + We have updated the health checks done by Seldon for the model nodes in your inference graph. If `executor.fullHealthChecks` is set to true then: * For Seldon protocol each node will be probed with `/api/v1.0/health/status`. - * For the Open Inference Protocol (or V2 protocol) each node will be probed with `/v2/health/ready`. * For tensorflow just TCP checks will be run on the http endpoint. + * For the Open Inference Protocol (or V2 protocol) each node will be probed with `/v2/health/ready`. By default we have set `executor.fullHealthChecks` to false for 1.14 release as users would need to rebuild their custom python models if they have not implemented the `health_status` method. In future we will default to `true`. diff --git a/examples/models/lightgbm_custom_server/iris.ipynb b/examples/models/lightgbm_custom_server/iris.ipynb index 6fec14f0b3..2802da08ef 100644 --- a/examples/models/lightgbm_custom_server/iris.ipynb +++ b/examples/models/lightgbm_custom_server/iris.ipynb @@ -7,9 +7,12 @@ "source": [ "# Custom LightGBM Prepackaged Model Server\n", "\n", + "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "\n", + "\n", "In this notebook we create a new custom LIGHTGBM_SERVER prepackaged server with two versions:\n", " * A Seldon protocol LightGBM model server\n", - " * A KfServing V2 protocol version using MLServer for running lightgbm models\n", + " * A KfServing V2 or Open Inference protocol version using MLServer for running lightgbm models\n", "\n", "The Seldon model server is in defined in `lightgbmserver` folder.\n", "\n", diff --git a/notebooks/backwards_compatability.ipynb b/notebooks/backwards_compatability.ipynb index 3f4b3fcc61..1d3916dd0f 100644 --- a/notebooks/backwards_compatability.ipynb +++ b/notebooks/backwards_compatability.ipynb @@ -12,7 +12,8 @@ " * curl\n", " * grpcurl\n", " * pygmentize\n", - " \n", + "\n", + "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. \n", "\n", "## Setup Seldon Core\n", "\n", diff --git a/notebooks/protocol_examples.ipynb b/notebooks/protocol_examples.ipynb index 289093555b..138fac62e4 100644 --- a/notebooks/protocol_examples.ipynb +++ b/notebooks/protocol_examples.ipynb @@ -15,6 +15,8 @@ " \n", "## Examples\n", "\n", + "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "\n", " * [Seldon Protocol](#Seldon-Protocol-Model)\n", " * [Tensorflow Protocol](#Tensorflow-Protocol-Model)\n", " * [V2 Protocol](#V2-Protocol-Model)\n", diff --git a/notebooks/server_examples.ipynb b/notebooks/server_examples.ipynb index cd0525789d..e38df79d6c 100644 --- a/notebooks/server_examples.ipynb +++ b/notebooks/server_examples.ipynb @@ -65,6 +65,8 @@ "source": [ "## Serve SKLearn Iris Model\n", "\n", + "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "\n", "In order to deploy SKLearn artifacts, we can leverage the [pre-packaged SKLearn inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html).\n", "The exposed API can follow either:\n", "\n", From 4ffeda7f0b7b6a0578542b45b9e7906fb27dbd9b Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:19:42 +0530 Subject: [PATCH 02/11] updated the note. --- doc/source/analytics/explainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/analytics/explainers.md b/doc/source/analytics/explainers.md index d929007d68..5b8b087df0 100644 --- a/doc/source/analytics/explainers.md +++ b/doc/source/analytics/explainers.md @@ -45,7 +45,7 @@ For an e2e example, please check AnchorTabular notebook [here](../examples/iris_ ## Explain API -**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. For the Seldon Protocol an endpoint path will be exposed for: From d2ff7e0b707d967890c52478d6448bf7f7f06491 Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:20:14 +0530 Subject: [PATCH 03/11] Update the note --- doc/source/graph/protocols.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/graph/protocols.md b/doc/source/graph/protocols.md index f237207566..64d26cd439 100644 --- a/doc/source/graph/protocols.md +++ b/doc/source/graph/protocols.md @@ -1,6 +1,6 @@ # Protocols -**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.. Tensorflow protocol is only available in version >=1.1. From 9e4afda799596bd51ca10363b7d3d3715ea7184c Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:20:43 +0530 Subject: [PATCH 04/11] Update optimization.md --- doc/source/production/optimization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/production/optimization.md b/doc/source/production/optimization.md index 6a874193c6..3fb3469a3a 100644 --- a/doc/source/production/optimization.md +++ b/doc/source/production/optimization.md @@ -9,7 +9,7 @@ Using the Seldon python wrapper there are various optimization areas one needs t ### Seldon Protocol Payload Types with REST and gRPC -**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. Depending on whether you want to use REST or gRPC and want to send tensor data the format of the request will have a deserialization/serialization cost in the python wrapper. This is investigated in a [python serialization notebook](../examples/python_serialization.html). From 5bcdac6353c350997991c1a715a959f51fb7a692 Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:21:14 +0530 Subject: [PATCH 05/11] Update upgrading.md --- doc/source/reference/upgrading.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/reference/upgrading.md b/doc/source/reference/upgrading.md index fd1ac436f0..e6f92c89df 100644 --- a/doc/source/reference/upgrading.md +++ b/doc/source/reference/upgrading.md @@ -93,7 +93,7 @@ Only the v1 versions of the CRD will be supported moving forward. The v1beta1 ve ### Model Health Checks -**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. We have updated the health checks done by Seldon for the model nodes in your inference graph. If `executor.fullHealthChecks` is set to true then: * For Seldon protocol each node will be probed with `/api/v1.0/health/status`. From 4ef43ad9c121427210e63601d9af9511a1e23f8c Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:21:41 +0530 Subject: [PATCH 06/11] Update iris.ipynb --- examples/models/lightgbm_custom_server/iris.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/models/lightgbm_custom_server/iris.ipynb b/examples/models/lightgbm_custom_server/iris.ipynb index 2802da08ef..3d81540ca8 100644 --- a/examples/models/lightgbm_custom_server/iris.ipynb +++ b/examples/models/lightgbm_custom_server/iris.ipynb @@ -7,7 +7,7 @@ "source": [ "# Custom LightGBM Prepackaged Model Server\n", "\n", - "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", "\n", "In this notebook we create a new custom LIGHTGBM_SERVER prepackaged server with two versions:\n", From 8e12e438dc29e9fe9a711a563d31c3a65ab6eed7 Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:22:14 +0530 Subject: [PATCH 07/11] Update backwards_compatability.ipynb --- notebooks/backwards_compatability.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/backwards_compatability.ipynb b/notebooks/backwards_compatability.ipynb index 1d3916dd0f..da6f884cad 100644 --- a/notebooks/backwards_compatability.ipynb +++ b/notebooks/backwards_compatability.ipynb @@ -13,7 +13,7 @@ " * grpcurl\n", " * pygmentize\n", "\n", - "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. \n", + "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. \n", "\n", "## Setup Seldon Core\n", "\n", From 3e3fec50c64e28ba9acbac2abc634414577ece28 Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:22:40 +0530 Subject: [PATCH 08/11] Update protocol_examples.ipynb --- notebooks/protocol_examples.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/protocol_examples.ipynb b/notebooks/protocol_examples.ipynb index 138fac62e4..b08ce036b0 100644 --- a/notebooks/protocol_examples.ipynb +++ b/notebooks/protocol_examples.ipynb @@ -15,7 +15,7 @@ " \n", "## Examples\n", "\n", - "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", " * [Seldon Protocol](#Seldon-Protocol-Model)\n", " * [Tensorflow Protocol](#Tensorflow-Protocol-Model)\n", From 68686ee64ce1b575e24b23ffe0205dfc7636bf2a Mon Sep 17 00:00:00 2001 From: Rajakavitha Kodhandapani Date: Mon, 6 Jan 2025 15:23:09 +0530 Subject: [PATCH 09/11] Update server_examples.ipynb --- notebooks/server_examples.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/server_examples.ipynb b/notebooks/server_examples.ipynb index e38df79d6c..c27b62f702 100644 --- a/notebooks/server_examples.ipynb +++ b/notebooks/server_examples.ipynb @@ -65,7 +65,7 @@ "source": [ "## Serve SKLearn Iris Model\n", "\n", - "**Note**: Seldon protocol and TensorFlow protocol are no longer supported, as Seldon has transitioned to the industry-standard Open Inference Protocol (OIP). Customers are encouraged to migrate to OIP, which offers seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", "In order to deploy SKLearn artifacts, we can leverage the [pre-packaged SKLearn inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html).\n", "The exposed API can follow either:\n", From 1babade845cf1f1ca9c6dbc3e1877ea6bbb89cea Mon Sep 17 00:00:00 2001 From: Rakavitha Kodhandapani Date: Tue, 7 Jan 2025 11:49:20 +0530 Subject: [PATCH 10/11] rephrased the wordings and made the OIP the default tab --- doc/source/analytics/explainers.md | 4 +- doc/source/graph/protocols.md | 76 ++++++++++--------- doc/source/production/optimization.md | 2 +- doc/source/reference/upgrading.md | 2 +- .../models/lightgbm_custom_server/iris.ipynb | 2 +- notebooks/backwards_compatability.ipynb | 2 +- notebooks/protocol_examples.ipynb | 5 +- notebooks/server_examples.ipynb | 6 +- 8 files changed, 52 insertions(+), 47 deletions(-) diff --git a/doc/source/analytics/explainers.md b/doc/source/analytics/explainers.md index 5b8b087df0..99dc2a731a 100644 --- a/doc/source/analytics/explainers.md +++ b/doc/source/analytics/explainers.md @@ -45,7 +45,7 @@ For an e2e example, please check AnchorTabular notebook [here](../examples/iris_ ## Explain API -**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. For the Seldon Protocol an endpoint path will be exposed for: @@ -89,9 +89,9 @@ The explain method is also supported for tensorflow and Open Inference protocols | Protocol | URI | | ------ | ----- | +| v2 | `http:////v2/models//infer` | | seldon | `http:////api/v1.0/explain` | | tensorflow | `http:////v1/models/:explain` | -| v2 | `http:////v2/models//infer` | Note: for `tensorflow` protocol we support similar non-standard extension as for the [prediction API](../graph/protocols.md#rest-and-grpc-tensorflow-protocol), `http:////v1/models/:explain`. diff --git a/doc/source/graph/protocols.md b/doc/source/graph/protocols.md index 64d26cd439..0c334f2375 100644 --- a/doc/source/graph/protocols.md +++ b/doc/source/graph/protocols.md @@ -1,48 +1,19 @@ # Protocols -**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.. +**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. Tensorflow protocol is only available in version >=1.1. Seldon Core supports the following data planes: - * [REST and gRPC Seldon protocol](#rest-and-grpc-seldon-protocol) - * [REST and gRPC Tensorflow Serving Protocol](#rest-and-grpc-tensorflow-protocol) - * [REST and gRPC Open Inference Protocol](#v2-protocol) + * REST and gRPC Open Inference Protocol + * REST and gRPC Seldon protocol + * REST and gRPC Tensorflow Serving Protocol -## REST and gRPC Seldon Protocol - * [REST Seldon Protocol](../reference/apis/index.html) - -Seldon is the default protocol for SeldonDeployment resources. You can specify the gRPC protocol by setting `transport: grpc` in your SeldonDeployment resource or ensuring all components in the graph have endpoint.tranport set ot grpc. - -See [example notebook](../examples/protocol_examples.html). - -## REST and gRPC Tensorflow Protocol - - * [REST Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md). - * [gRPC Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto). - -Activate this protocol by speicfying `protocol: tensorflow` and `transport: rest` or `transport: grpc` in your Seldon Deployment. See [example notebook](../examples/protocol_examples.html). - -For Seldon graphs the protocol will work as expected for single model graphs for Tensorflow Serving servers running as the single model in the graph. For more complex graphs you can chain models: - - * Sending the response from the first as a request to the second. This will be done automatically when you defined a chain of models as a Seldon graph. It is up to the user to ensure the response of each changed model can be fed a request to the next in the chain. - * Only Predict calls can be handled in multiple model chaining. +{% tabs %} - -General considerations: - - * Seldon components marked as MODELS, INPUT_TRANSFORMER and OUTPUT_TRANSFORMERS will allow a PredictionService Predict method to be called. - * GetModelStatus for any model in the graph is available. - * GetModelMetadata for any model in the graph is available. - * Combining and Routing with the Tensorflow protocol is not presently supported. - * `status` and `metadata` calls can be asked for any model in the graph - * a non-standard Seldon extension is available to call predict on the graph as a whole: `/v1/models/:predict`. - * The name of the model in the `graph` section of the SeldonDeployment spec must match the name of the model loaded onto the Tensorflow Server. - - -## Open Inference Protocol (or V2 protocol) +{% tab title="REST and gRPC Open Inference Protocol" %} Seldon has collaborated with the [NVIDIA Triton Server Project](https://github.com/triton-inference-server/server) and the [KServe @@ -89,3 +60,38 @@ In particular, | [MLFLOW_SERVER](../servers/mlflow.md) | ✅ | [Seldon MLServer](https://github.com/seldonio/mlserver) | You can try out the `v2` in [this example notebook](../examples/protocol_examples.html). + {% endtab %} + +{% tab title="REST and gRPC Seldon Protocol" %} +* [REST Seldon Protocol](../reference/apis/index.html) + +Seldon is the default protocol for SeldonDeployment resources. You can specify the gRPC protocol by setting `transport: grpc` in your SeldonDeployment resource or ensuring all components in the graph have endpoint.tranport set ot grpc. + +See [example notebook](../examples/protocol_examples.html). +{% endtab %} + +{% tab title="REST and gRPC Tensorflow Protocol" %} + * [REST Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md). + * [gRPC Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto). + +Activate this protocol by speicfying `protocol: tensorflow` and `transport: rest` or `transport: grpc` in your Seldon Deployment. See [example notebook](../examples/protocol_examples.html). + +For Seldon graphs the protocol will work as expected for single model graphs for Tensorflow Serving servers running as the single model in the graph. For more complex graphs you can chain models: + + * Sending the response from the first as a request to the second. This will be done automatically when you defined a chain of models as a Seldon graph. It is up to the user to ensure the response of each changed model can be fed a request to the next in the chain. + * Only Predict calls can be handled in multiple model chaining. + + +General considerations: + + * Seldon components marked as MODELS, INPUT_TRANSFORMER and OUTPUT_TRANSFORMERS will allow a PredictionService Predict method to be called. + * GetModelStatus for any model in the graph is available. + * GetModelMetadata for any model in the graph is available. + * Combining and Routing with the Tensorflow protocol is not presently supported. + * `status` and `metadata` calls can be asked for any model in the graph + * a non-standard Seldon extension is available to call predict on the graph as a whole: `/v1/models/:predict`. + * The name of the model in the `graph` section of the SeldonDeployment spec must match the name of the model loaded onto the Tensorflow Server. {% endtab %} + +{% endtabs %} + + diff --git a/doc/source/production/optimization.md b/doc/source/production/optimization.md index 3fb3469a3a..962efa311e 100644 --- a/doc/source/production/optimization.md +++ b/doc/source/production/optimization.md @@ -9,7 +9,7 @@ Using the Seldon python wrapper there are various optimization areas one needs t ### Seldon Protocol Payload Types with REST and gRPC -**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. Depending on whether you want to use REST or gRPC and want to send tensor data the format of the request will have a deserialization/serialization cost in the python wrapper. This is investigated in a [python serialization notebook](../examples/python_serialization.html). diff --git a/doc/source/reference/upgrading.md b/doc/source/reference/upgrading.md index e6f92c89df..578cdd8760 100644 --- a/doc/source/reference/upgrading.md +++ b/doc/source/reference/upgrading.md @@ -93,7 +93,7 @@ Only the v1 versions of the CRD will be supported moving forward. The v1beta1 ve ### Model Health Checks -**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. +**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. We have updated the health checks done by Seldon for the model nodes in your inference graph. If `executor.fullHealthChecks` is set to true then: * For Seldon protocol each node will be probed with `/api/v1.0/health/status`. diff --git a/examples/models/lightgbm_custom_server/iris.ipynb b/examples/models/lightgbm_custom_server/iris.ipynb index 3d81540ca8..d0db169fc5 100644 --- a/examples/models/lightgbm_custom_server/iris.ipynb +++ b/examples/models/lightgbm_custom_server/iris.ipynb @@ -7,7 +7,7 @@ "source": [ "# Custom LightGBM Prepackaged Model Server\n", "\n", - "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", "\n", "In this notebook we create a new custom LIGHTGBM_SERVER prepackaged server with two versions:\n", diff --git a/notebooks/backwards_compatability.ipynb b/notebooks/backwards_compatability.ipynb index da6f884cad..ea6e94596b 100644 --- a/notebooks/backwards_compatability.ipynb +++ b/notebooks/backwards_compatability.ipynb @@ -13,7 +13,7 @@ " * grpcurl\n", " * pygmentize\n", "\n", - "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience. \n", + "**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", "## Setup Seldon Core\n", "\n", diff --git a/notebooks/protocol_examples.ipynb b/notebooks/protocol_examples.ipynb index b08ce036b0..cb5b41b438 100644 --- a/notebooks/protocol_examples.ipynb +++ b/notebooks/protocol_examples.ipynb @@ -15,11 +15,10 @@ " \n", "## Examples\n", "\n", - "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", - "\n", + "**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + " * [Open Inference Protocol or V2 Protocol](#V2-Protocol-Model)\n", " * [Seldon Protocol](#Seldon-Protocol-Model)\n", " * [Tensorflow Protocol](#Tensorflow-Protocol-Model)\n", - " * [V2 Protocol](#V2-Protocol-Model)\n", " \n", "\n", "## Setup Seldon Core\n", diff --git a/notebooks/server_examples.ipynb b/notebooks/server_examples.ipynb index c27b62f702..3ce53573a9 100644 --- a/notebooks/server_examples.ipynb +++ b/notebooks/server_examples.ipynb @@ -65,13 +65,13 @@ "source": [ "## Serve SKLearn Iris Model\n", "\n", - "**Note**: Seldon is phasing out support for the Seldon and TensorFlow protocols, transitioning to the industry-standard Open Inference Protocol (OIP). We encourage customers to migrate to OIP, which facilitates seamless integration across various model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", + "**Note**: Seldon is no longer maintaining the Seldon and TensorFlow protocols. Instead, Seldon is adopting the industry-standard Open Inference Protocol (OIP). We strongly encourage customers to use the OIP, which offers seamless integration across diverse model serving runtimes, supports the creation of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.\n", "\n", "In order to deploy SKLearn artifacts, we can leverage the [pre-packaged SKLearn inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html).\n", "The exposed API can follow either:\n", + "- Open Inference Protocol or V2 Protocol.\n", + "- Seldon protocol. \n", "\n", - "- The default Seldon protocol. \n", - "- The V2 protocol.\n", "\n" ] }, From 25d0c5525f379cb57dcc68f33ac5ac834b3d0779 Mon Sep 17 00:00:00 2001 From: Rakavitha Kodhandapani Date: Tue, 7 Jan 2025 11:54:48 +0530 Subject: [PATCH 11/11] removed the tabs --- doc/source/graph/protocols.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/doc/source/graph/protocols.md b/doc/source/graph/protocols.md index 0c334f2375..694f3f3a84 100644 --- a/doc/source/graph/protocols.md +++ b/doc/source/graph/protocols.md @@ -11,9 +11,8 @@ Seldon Core supports the following data planes: * REST and gRPC Tensorflow Serving Protocol -{% tabs %} -{% tab title="REST and gRPC Open Inference Protocol" %} +### REST and gRPC Open Inference Protocol Seldon has collaborated with the [NVIDIA Triton Server Project](https://github.com/triton-inference-server/server) and the [KServe @@ -60,17 +59,16 @@ In particular, | [MLFLOW_SERVER](../servers/mlflow.md) | ✅ | [Seldon MLServer](https://github.com/seldonio/mlserver) | You can try out the `v2` in [this example notebook](../examples/protocol_examples.html). - {% endtab %} -{% tab title="REST and gRPC Seldon Protocol" %} + +### REST and gRPC Seldon Protocol * [REST Seldon Protocol](../reference/apis/index.html) Seldon is the default protocol for SeldonDeployment resources. You can specify the gRPC protocol by setting `transport: grpc` in your SeldonDeployment resource or ensuring all components in the graph have endpoint.tranport set ot grpc. See [example notebook](../examples/protocol_examples.html). -{% endtab %} -{% tab title="REST and gRPC Tensorflow Protocol" %} +### REST and gRPC Tensorflow Protocol * [REST Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md). * [gRPC Tensorflow Protocol definition](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto). @@ -92,6 +90,5 @@ General considerations: * a non-standard Seldon extension is available to call predict on the graph as a whole: `/v1/models/:predict`. * The name of the model in the `graph` section of the SeldonDeployment spec must match the name of the model loaded onto the Tensorflow Server. {% endtab %} -{% endtabs %}