Sending Feedback to Models

docs/api-testing.md

@@ -0,0 +1,152 @@
+# Testing Your Seldon Components
+
+Whether you have wrapped your component using [our S2I wrappers](./wrappers/readme.md) or created your own wrapper you will want to test the Docker container standalone and also quickly within a running cluster. We have provided two python tools to allow you to easily do this
+
+ * [Microservice API Tester](../util/api_tester)
+    * Allows you to test a docker component to check it respects the Seldon API
+ * [Seldon-Core API Tester](../wrappers/testing)
+    * Allows you to test the external endpoints for a running Seldon Deployment graph.
+
+
+## Microservice API Tester
+
+Use this [script](../util/api_tester/api-tester.py) to test a Seldon graph deployed to a cluster.
+
+```
+usage: api-tester.py [-h] [--endpoint {predict,send-feedback}] [-b BATCH_SIZE]
+                     [-n N_REQUESTS] [--grpc] [-t] [-p]
+                     [--oauth-port OAUTH_PORT] [--oauth-key OAUTH_KEY]
+                     [--oauth-secret OAUTH_SECRET]
+                     [--ambassador-path AMBASSADOR_PATH]
+                     contract host port
+
+positional arguments:
+  contract              File that contains the data contract
+  host
+  port
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --endpoint {predict,send-feedback}
+  -b BATCH_SIZE, --batch-size BATCH_SIZE
+  -n N_REQUESTS, --n-requests N_REQUESTS
+  --grpc
+  -t, --tensor
+  -p, --prnt            Prints requests and responses
+  --oauth-port OAUTH_PORT
+  --oauth-key OAUTH_KEY
+  --oauth-secret OAUTH_SECRET
+  --ambassador-path AMBASSADOR_PATH
+
+```
+
+Example:
+
+```
+python api-tester.py contract.json  0.0.0.0 8003 --oauth-key oauth-key --oauth-secret oauth-secret -p --grpc --oauth-port 8002 --endpoint send-feedback
+```
+
+ The above sends a gRPC send-feedback request to 0.0.0.0:8003 using the given oauth key/secret (assumes you are using the Seldon API Gateway) with the REST oauth-port at 8002 and use the contract.json file to create a random request. In this example you would have port-forwarded the Seldon api-server to local ports.
+
+You can find more exampes in the [example models folder notebooks](../examples/models).
+
+## Seldon API Tester
+
+Use this [script](../wrappers/testing/tester.py) to test a packaged Docker microservice Seldon component.
+
+```
+usage: tester.py [-h] [--endpoint {predict,send-feedback}] [-b BATCH_SIZE]
+                 [-n N_REQUESTS] [--grpc] [--fbs] [-t] [-p]
+                 contract host port
+
+positional arguments:
+  contract              File that contains the data contract
+  host
+  port
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --endpoint {predict,send-feedback}
+  -b BATCH_SIZE, --batch-size BATCH_SIZE
+  -n N_REQUESTS, --n-requests N_REQUESTS
+  --grpc
+  --fbs
+  -t, --tensor
+  -p, --prnt            Prints requests and responses
+```
+
+Example:
+
+```
+python tester.py contract.json 0.0.0.0 5000 -p --grpc
+```
+
+The above sends a predict call to a gRPC component exposed at 0.0.0.0:5000 using the contract.json to create a random request.
+
+You can find more exampes in the [example models folder notebooks](../examples/models).
+
+## API Contract
+
+Both test programs require you to provide a contract.json file defining the data you intend to send in a request and the response you expect back.
+
+An example for the example Iris classification model is shown below:
+
+```
+{
+    "features":[
+	{
+	    "name":"sepal_length",
+	    "dtype":"FLOAT",
+	    "ftype":"continuous",
+	    "range":[4,8]
+	},
+	{
+	    "name":"sepal_width",
+	    "dtype":"FLOAT",
+	    "ftype":"continuous",
+	    "range":[2,5]
+	},
+	{
+	    "name":"petal_length",
+	    "dtype":"FLOAT",
+	    "ftype":"continuous",
+	    "range":[1,10]
+	},
+	{
+	    "name":"petal_width",
+	    "dtype":"FLOAT",
+	    "ftype":"continuous",
+	    "range":[0,3]
+	}
+    ],
+    "targets":[
+	{
+	    "name":"class",
+	    "dtype":"FLOAT",
+	    "ftype":"continuous",
+	    "range":[0,1],
+	    "repeat":3
+	}
+    ]
+}
+```
+
+Here we have 4 input features each of which is continuous in certain ranges. The response targets will be a repeated set of floats in the 0-1 range.
+
+### Definition
+
+There are two sections:
+
+ * ```features``` : The types of the feature array that will be in the request
+ * ```targets``` : The types of the feature array that will be in the response
+
+Each section has a list of definitions. Each definition consists of:
+
+  * ```name``` : String : The name of the feature
+  * ```ftype``` : one of CONTINUOUS, CATEGORICAL : the type of the feature
+  * ```dtype``` : One of FLOAT, INT : Required for ftype CONTINUOUS : What type of feature to create
+  * ```values``` : list of Strings : Required for ftype CATEGORICAL : The possible categorical values
+  * ```range``` : list of two numbers : Optional for ftype CONTINUOUS : The range of values (inclusive) that a continuous value can take
+  * ```repeat``` : integer : Optional value for how many times to repeat this value
+  * ```shape``` : array of integers : Optional value for the shape of array to coerce the values
+

docs/wrappers/readme.md

@@ -7,6 +7,8 @@ To allow your component (model, router etc.) to be managed by seldon-core it nee
 
 To wrap your model follow the instructions for your chosen language or toolkit.
 
+To test a wrapped components you can use one of our [testing scripts](../api-testing.md).
+
 ## Python
 
 Python based models, including [TensorFlow](https://www.tensorflow.org/), [Keras](https://keras.io/), [pyTorch](http://pytorch.org/), [StatsModels](http://www.statsmodels.org/stable/index.html), [XGBoost](https://github.com/dmlc/xgboost) and [Scikit-learn](http://scikit-learn.org/stable/) based models.

engine/src/main/java/io/seldon/engine/predictors/PredictorConfigBean.java

@@ -47,6 +47,7 @@ public PredictorConfigBean(
         // MODEL -> TRANSFORM INPUT
         List<PredictiveUnitMethod> modelMethods = new ArrayList<PredictiveUnitMethod>();
         modelMethods.add(PredictiveUnitMethod.TRANSFORM_INPUT);
+        modelMethods.add(PredictiveUnitMethod.SEND_FEEDBACK);        
         typeMethodsMap.put(PredictiveUnitType.MODEL, modelMethods);
 
         // TRANSFORMER -> TRANSFORM INPUT

engine/src/main/java/io/seldon/engine/service/InternalPredictionService.java

@@ -195,6 +195,14 @@ public SeldonMessage sendFeedback(Feedback feedback, PredictiveUnitState state)
 							.withMaxOutboundMessageSize(grpcMaxMessageSize);
 					return stub.sendFeedback(feedback);
 				}
+				else if (state.type == PredictiveUnitType.MODEL)
+				{
+					ModelBlockingStub modelStub = ModelGrpc.newBlockingStub(getChannel(endpoint))
+							.withDeadlineAfter(grpcReadTimeout, TimeUnit.MILLISECONDS)
+							.withMaxInboundMessageSize(grpcMaxMessageSize)
+							.withMaxOutboundMessageSize(grpcMaxMessageSize);
+						return modelStub.sendFeedback(feedback);
+				}
 				else {
 					RouterBlockingStub routerStub =  RouterGrpc.newBlockingStub(getChannel(endpoint))
 							.withDeadlineAfter(grpcReadTimeout, TimeUnit.MILLISECONDS)

examples/models/README.md

@@ -0,0 +1,21 @@
+# Example Models
+
+These examples provide illustrations of creating various wrapped ML models for deployment using Seldon.
+
+   * Python
+      * [Tensorflow MNIST Classifier](./deep_mnist/deep_mnist.ipynb)
+      * [Keras MNIST Classifier](./keras_mnist/keras_mnist.ipynb)
+      * [Scikit-learn MNIST Classifier](./sk_mnist/skmnist.ipynb)
+      * [Scikit-learn Iris Classifier](./sklearn_iris/sklearn_iris.ipynb)
+   * R
+      * [R MNIST Classifier](./r_mnist/r_mnist.ipynb)
+      * [R Iris Classifier](./r_iris/r_iris.ipynb)
+   * Java
+      * [H2O Classifier](./h2o_mojo/h2o_model.ipynb)
+   * NodeJS
+      * [Tensorflow MNIST Classifier](./nodejs_mnist/nodejs_mnist.ipynb)
+   * ONNX
+      * [ResNet ONNX Classifier using Intel nGraph](./onnx_resnet50/onnx_resnet50.ipynb)
+   * Misc
+      * Custom endpoints (for Prometheus)
+

examples/models/deep_mnist/deep_mnist.ipynb

@@ -129,6 +129,15 @@
     "!docker rm mnist_predictor --force"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Test using Minikube\n",
+    "\n",
+    "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need Minikube version 0.25.2**"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -156,6 +165,15 @@
     "!helm init"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

examples/models/h2o_mojo/h2o_model.ipynb

@@ -123,6 +123,15 @@
     "!docker rm h2o_predictor --force"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Test using Minikube\n",
+    "\n",
+    "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need Minikube version 0.25.2**"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -150,6 +159,15 @@
     "!helm init"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

examples/models/keras_mnist/keras_mnist.ipynb

@@ -297,6 +297,15 @@
     "!docker rm mnist_predictor --force"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Test using Minikube\n",
+    "\n",
+    "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need Minikube version 0.25.2**"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -324,6 +333,15 @@
     "!helm init"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

examples/models/nodejs_mnist/nodejs_mnist.ipynb

@@ -160,7 +160,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Test on Minikube"
+    "# Test using Minikube\n",
+    "\n",
+    "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need Minikube version 0.25.2**"
    ]
   },
   {
@@ -190,6 +192,15 @@
     "!helm init"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,