diff --git a/.gitignore b/.gitignore index 63df756c70..19f164c70e 100644 --- a/.gitignore +++ b/.gitignore @@ -143,3 +143,9 @@ openapi/tensorflow/ proto/tensorflow/tensorflow/ util/loadtester/MNIST_data/ + +# python build +eggs/ +.eggs/ +*.egg-info/ +./pytest_cache diff --git a/docs/examples/h2oexample.md b/docs/examples/h2oexample.md deleted file mode 100644 index 16f6103094..0000000000 --- a/docs/examples/h2oexample.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: "How to wrap a h2o model" -date: 2017-12-09T17:49:41Z ---- - - -In this readme we outline the steps needed to wrap any h2o model using seldon python wrappers into a docker image deployable with seldon core. - -The file H2oModel.py is a template to use in order to wrap a h2o model. The only modification required on the user side consists of setting the MODEL_PATH variable at the top of the file as explained below in session "Wrap" at point 2. - -We also provide a script to train and save a pre-built h2o model predicting bad loans. - -The session "General use" explain how to use the wrapper with any saved h2o model. - -The session "Example of usage" provides a step-by-step guide for training, deploying and wrap the pre-built h2o model for bad loans predictions as an example. - -## General use - -### Preliminary steps - -1. It is assumed you have already trained a h2o model and saved it in a file \ using the ```h2o.save_model()``` python function. -* You have cloned the latest version of seldon-core git repository. -* You have a python+java docker image available to use as base-image. One way to build a suitable base-image locally is by using the [Dockerfile provided by h2o](https://h2o-release.s3.amazonaws.com/h2o/rel-turing/1/docs-website/h2o-docs/docker.html): - * Make sure you have docker daemon running. - * Download the [Dockerfile provided by h2o](https://h2o-release.s3.amazonaws.com/h2o/rel-turing/1/docs-website/h2o-docs/docker.html) in any folder. - * Create the base docker image: - - ``` docker build --force-rm=true -t .``` - - Building the image may take several minutes. - -### Wrap: - -You can now wrap the model using seldon python wrappers. - -1. Copy the files H2oModel.py, requirements.txt and \ in a folder named \ . -* Open the file H2oModel.py with your favourite editor and set the variable MODEL_PATH to: - - ```MODEL_PATH=/microservice/``` -* Cd into the the python wrapper directory in seldon-core, ```/seldon-core/wrapper/python``` -* Use the wrap_model.py script to wrap your model: - - ```python wrap_model.py H2oModel --base-image --force``` - - This will create a "build" directory in \. -* Enter the build directory created by wrap_model.py: - - ```cd /build``` -* Build the docker image of your model: - - ```make build_docker_image``` - - This will build a docker image named ```/h2omodel:```. - -### Deploy in seldon-core: - -You can now deploy the model as a docker image ```/h2omodel:``` using [seldon-core](../../api/seldon-deployment). - - - -## Example of usage - -Here we give an example of usage step by step in which we will train and save a [h2o model for bad loan predictions](https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/intro-to-h2o.ipynb), we will create a base image supporting h2o named "none/h2obase:0.0" and we will use seldon wrappers to build dockererized version of the model ready to be deployed with seldon-core. - -### Preliminary step: build your base image locally - -1. Make sure you have docker daemon running -* Download the [Dockerfile provided by h2o](https://h2o-release.s3.amazonaws.com/h2o/rel-turing/1/docs-website/h2o-docs/docker.html) in any directory. -* Run ``` docker build --force-rm=true -t none/h2obase:0.0 .``` in the same directory. This will create the base image "none/h2obase:0.0" locally (may take several minutes). - -### Train and wrap the model - -1. Clone seldon-core and seldon-core-plugins git repositories in the same directory. -* Train and saved the model: - - ```cd seldon-core-plugins/h2o_example``` - - ```python train_model.py``` - - This will train the model and save it in a file named "glm_fit1"" in the same directory. -* Wrap the model: - - ```cd ../../seldon-core/wrappers/python``` - - ```python wrap_model.py ../../../seldon-core-plugins/h2o_example H2oModel 0.0 none --base-image none/h2obase:0.0 --force``` - - This will create a directory "build" in "seldon-core-plugins/h2o_example". -* Create docker image: - - ```cd ../../../seldon-core-plugins/h2o_example/build``` - - ```make build_docker_image``` - - This will create a docker image "none/h2omodel:0.0", which is ready to be deployed in seldon-core. - -Note that the steps 3 and 4 are equivalent to steps 3-6 in the general use session above. diff --git a/docs/wrappers/h2o.md b/docs/wrappers/h2o.md deleted file mode 100644 index 499403d2cc..0000000000 --- a/docs/wrappers/h2o.md +++ /dev/null @@ -1,94 +0,0 @@ -# Packaging a H2O model for Seldon Core - -This document outlines the steps needed to wrap any H2O model using Seldon's python wrappers into a docker image ready for deployment with Seldon Core. The process is nearly identical to [wrapping a python model with the seldon wrapper](python-docker.md), so be sure to read the documentation on this process first. -The main differences are: -* The data sent to the model needs to be transformed from numpy arrays into H2O Frames and back; -* The base docker image has to be changed, because H2O needs a Java Virtual Machine installed in the container. - -You will find below explanations for: -* How to build the H2O base docker image; -* How to wrap your H2O model; -* A detailed example where we train and wrap a bad loans prediction model. - -## Building the H2O base docker image - -In order to wrap a H2O model with the python wrappers, you need a python+java docker image available to use as base image. One way to build a suitable base image locally is by using the [Dockerfile provided by H2O](https://h2o-release.s3.amazonaws.com/h2o/rel-turing/1/docs-website/h2o-docs/docker.html): - -* Make sure you have docker daemon running. -* Download the [Dockerfile provided by H2O](https://github.com/h2oai/h2o-3/blob/master/Dockerfile) in any folder. -* Create the base docker image (we will call it H2OBase:1.0 in this example): - - ```bash - docker build --force-rm=true -t H2OBase:1.0 . - ``` - -Building the image may take several minutes. - -## Wrapping the model - - -It is assumed you have already trained a H2O model and saved it in a file (called in what follows SavedModel.h2o). If you use the H2O python API, you can save your model using the```h2o.save_model()``` method. - -You can now wrap the model using Seldon's python wrappers. This is similar to the general python model wrapping process except that you need to specify the H2O base image as an argument when calling the wrapping script. - -We provide a file [H2OModel.py](https://github.com/SeldonIO/seldon-core/blob/master/examples/models/h2o_example/H2OModel.py) as a template for the model entry point, which handles loading the H2OModel and transforming the data between numpy and H2O Frames. In what follows we assume you are using this template. The H2O model is loaded in the class constructor and the numpy arrays are turned into H2O Frames when received in the predict method. - -Detailed steps: -1. Put the files H2OModel.py, requirements.txt and SavedModel.h2o in a directory created for this purpose. -2. Open the file H2OModel.py with your favourite text editor and set the variable MODEL_PATH to: - - ```python - MODEL_PATH=./SavedModel.h2o - ``` - -3. Run the python wrapping scripts, with the additional ````--base-image``` argument: - - ```bash - docker run -v /path/to/your/model/folder:/model seldonio/core-python-wrapper:0.7 /model H2OModel 0.1 myrepo --base-image=H2OBase:1.0 - ``` - - "0.1" is the version of the docker image that will be created. "myrepo" is the name of your dockerhub repository. - -4. CD into the newly generated "build" directory and run: - - ```bash - ./build_image.sh - ./push_image.sh - ``` - - This will build and push to dockerhub a docker image named ```myrepo/h2omodel:0.1``` which is ready for deployment in seldon-core. - -## Example - -Here we give a step by step example in which we will train and save a [H2O model for bad loans predictions](https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/intro-to-h2o.ipynb), before turning it into a dockerized microservice. - -### Preliminary Requirements - -1. Have [H2O](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html) installed on your machine (H2O is only required to train the example. Seldon Core and Seldon wrappers do not require H2O installed on your machine) -2. You need to have built the base H2O docker image (see the [dedicated section](#building-the-h2o-base-docker-image) above) - -### Train and wrap the model - -1. Clone the Seldon Core git repository - - ```bash - git clone https://github.com/SeldonIO/seldon-core - ``` - -2. Train and save the H2O model for bad loans prediction: - - ```bash - cd seldon-core/examples/models/h2o_example/ - python train_model.py - ``` - - This will train the model and save it in a file named "glm_fit1"" in the same directory. - -3. Wrap the model: - - ```bash - cd ../../ - docker run -v models/h2o_example:my_model seldonio/core-python-wrapper:0.7 my_model H2OModel 0.1 myrepo --base-image=H2OBase:1.0 - ``` - - This will create a docker image "seldonio/h2omodel:0.1", which is ready to be deployed in seldon-core. diff --git a/docs/wrappers/nodejs.md b/docs/wrappers/nodejs.md index d21ee97a48..049de348c7 100644 --- a/docs/wrappers/nodejs.md +++ b/docs/wrappers/nodejs.md @@ -15,7 +15,7 @@ If you are not familiar with s2i you can read [general instructions on using s2i To check everything is working you can run ```bash -s2i usage seldonio/seldon-core-s2i-r:0.1 +s2i usage seldonio/seldon-core-s2i-nodejs:0.1 ``` # Step 2 - Create your source code diff --git a/docs/wrappers/python-docker.md b/docs/wrappers/python-docker.md deleted file mode 100644 index 7bfd2f415b..0000000000 --- a/docs/wrappers/python-docker.md +++ /dev/null @@ -1,134 +0,0 @@ -# Packaging a python model for Seldon Core using Seldon Wrapper -In this guide, we illustrate the steps needed to wrap your own python model in a docker image ready for deployment with Seldon Core, using the Seldon wrapper script. - -We suggest you look at using the [S2I tool for python models](python.md) before choosing this method. - -This script is designed to take your python model and turn it into a dockerised microservice that conforms to Seldon's internal API, thus avoiding the hassle to write your own dockerised microservice. - -You can use these wrappers with any model that offers a python API. Some examples are: - - * [TensorFlow](https://www.tensorflow.org/) - * [Keras](https://keras.io/) - * [pyTorch](http://pytorch.org/) - * [StatsModels](http://www.statsmodels.org/stable/index.html) - * [Scikit-learn](http://scikit-learn.org/stable/) - * [XGBoost](https://github.com/dmlc/xgboost) - -The global process is as follows: -* Regroup your files under a single directory and create a standard python class that will be used as an entry point -* Run the wrapper script that will package your model for docker -* Build and publish a docker image from the generated files - - -## Create a model folder - -The Seldon python wrappers are designed to take your model and turn it into a dockerised microservice that conforms to Seldon's internal API. -To wrap a model, there are 2 requirements: -* All the files that will be used at runtime need to be put into a single directory; -* You need a file that contains a standardised python class that will serve as an entry point for runtime predictions. - -Additionally, if you are making use of specific python libraries, you need to list them in a requirements.txt file that will be used by pip to install the packages in the docker image. - -Here we illustrate the content of the ```keras_mnist``` model folder which can be found in [seldon-core/examples/models/](https://github.com/SeldonIO/seldon-core/tree/master/examples). - -This folder contains the following 3 files: - -1. MnistClassifier.py: This is the entry point for the model. It needs to include a python class having the same name as the file, in this case MnistClassifier, that implements a method called predict that takes as arguments a multi-dimensional numpy array (X) and a list of strings (feature_names), and returns a numpy array of predictions. - - - ```python - from keras.models import load_model - - class MnistClassifier(object): # The file is called MnistClassifier.py - - def __init__(self): - """ You can load your pre-trained model in here. The instance will be created once when the docker container starts running on the cluster. """ - self.model = load_model('MnistClassifier.h5') - - def predict(self,X,feature_names): - """ X is a 2-dimensional numpy array, feature_names is a list of strings. - This methods needs to return a numpy array of predictions.""" - return self.model.predict(X) - ``` - -2. requirements.txt: List of the packages required by your model, that will be installed via ```pip install```. - - ``` - keras==2.0.6 - h5py==2.7.0 - ``` - -3. MnistClassifier.h5: This hdf file contains the saved keras model. - -## Wrap the model - -After you have copied the required files in your model folder, you run the Seldon wrapper script to turn your model into a dockerised microservice. The wrapper script requires as arguments the path to your model directory, the model name, a version for the docker image, and the name of a docker repository. It will generate a "build" directory that contains the microservice, Dockerfile, etc. - -In order to make things even simpler (and because we love Docker!) we have dockerised the wrapper script so that you don't need to install anything on your machine to run it - except Docker. - -``` -docker run -v /path/to/model/dir:/my_model seldonio/core-python-wrapper:0.7 /my_model MnistClassifier 0.1 seldonio -``` - -Let's explain each piece of this command in more details. - - -``` docker run seldonio/core-python-wrapper:0.7 ``` : run the core-python-wrapper container. - -``` -v /path/to/model/dir:/my_model ``` : Tells docker to mount your local folder to /my_model in the container. This is used to access your files and generate the wrapped model files. - -``` /my_model MnistClassifier 0.1 seldonio ``` : These are the command line arguments that are passed to the script. The bare minimum, as in this example, are the path where your model folder has been mounted in the container, the model name, the docker image version and the docker hub repository. - -For reference, here is the complete list of arguments that can be passed to the script. - -``` -docker run -v /path: seldonio/core-python-wrapper:0.7 - - - - - --out-folder= - --service-type= - --base-image= - --image-name= - --force - --persistence - --grpc -``` - -Required: -* model_path: The path to the model folder inside the container - the same as the mount you have chosen, in our above example my_model -* model_name: The name of your model class and file, as defined above. In our example, MnistClassifier -* image_version: The version of the docker image you will create. By default the name of the image will be the name of your model in lower-case (more on how to change this later). In our example 0.1 -* docker_repo: The name of your dockerhub repository. In our example seldonio. - -Optional: -* out-folder: The folder that will be created to contain the output files. Defaults to ./build -* service-type: The type of Seldon Service API the model will use. Defaults to MODEL. Other options are ROUTER, COMBINER, TRANSFORMER, OUTPUT_TRANSFORMER -* base-image: The docker image your docker container will inherit from. Defaults to python:2. -* image-name: The name of your docker image. Defaults to model_name in lower-case -* force: When this flag is present, the build folder will be overwritten if it already exists. The wrapping is aborted by default. -* persistence: When this flag is present, the model will be made persistent, its state being saved at a regular interval on redis. -* grpc: When this flag is present, the model will expose a GRPC API rather than the default REST API - -Note that you can access the command line help of the script by using the -h or --help argument as follows: - -``` -docker run seldonio/core-python-wrapper:0.7 -h -``` - -Note also that you could use the python script directly if you feel so inclined, but you would have to check out seldon-core and install some python libraries on your local machine - by using the docker image you don't have to care about these dependencies. - -## Build and push the Docker image - -A folder named "build" should have appeared in your model directory. It contains all the files needed to build and publish your model's docker image. - -To do so, run: - -``` -cd /path/to/model/dir/build -./build_image.sh -./push_image.sh -``` - -And voilà, the docker image for your model is now available in your docker repository, and Seldon Core can deploy it into production. diff --git a/docs/wrappers/readme.md b/docs/wrappers/readme.md index 4e03cce552..a84d3e371b 100644 --- a/docs/wrappers/readme.md +++ b/docs/wrappers/readme.md @@ -13,14 +13,11 @@ To test a wrapped components you can use one of our [testing scripts](../api-tes Python based models, including [TensorFlow](https://www.tensorflow.org/), [Keras](https://keras.io/), [pyTorch](http://pytorch.org/), [StatsModels](http://www.statsmodels.org/stable/index.html), [XGBoost](https://github.com/dmlc/xgboost) and [Scikit-learn](http://scikit-learn.org/stable/) based models. -You can use either: - -- [Source-to-image (s2i) tool](./python.md) -- [Seldon Docker wrapper application](./python-docker.md) +- [Python models wrapped using source-to-image](./python.md) ## R -- [R models can be wrapped using source-to-image](r.md) +- [R models wrapped using source-to-image](r.md) ## Java @@ -28,13 +25,6 @@ Java based models including, [H2O](https://www.h2o.ai/), [Deep Learning 4J](http - [Java models wrapped using source-to-image](java.md) -## H2O - -H2O models can be wrapped either from Java or Python. - -- [Java models wrapped using source-to-image](java.md) -- [H2O models saved and called from python](./h2o.md) - ## NodeJS -- [Javascript models can be wrapped using source-to-image](nodejs.md) +- [Javascript models wrapped using source-to-image](nodejs.md) diff --git a/docs/wrappers/s2i.md b/docs/wrappers/s2i.md index 686f013c88..79a7aa9023 100644 --- a/docs/wrappers/s2i.md +++ b/docs/wrappers/s2i.md @@ -16,7 +16,8 @@ The general work flow is: At present we have s2i builder images for - * [python (python2 or python3)](./python.md) : use this for Tensorflow, Keras, pyTorch or sklearn models. + * [Python (Python2 or Python3)](./python.md) : use this for Tensorflow, Keras, PyTorch or sklearn models. * [R](r.md) * [Java](java.md) + * [NodeJS](nodejs.md) diff --git a/examples/README.md b/examples/README.md index 7e9dc486f6..9a5d62aab8 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,12 +2,6 @@ Seldon-core-examples repository provides out-of-the-box machine learning models examples to deploy using [seldon-core](https://github.com/SeldonIO/seldon-core). Since seldon-core deploys dockerized versions of your models, the repository also includes wrapping scripts that allow you to create docker images of such models which are deployable with seldon-core. -## Wrapping scripts - -The repository contains two wrapping scripts at the moment -* wrap-model-in-host : If you are using docker on your machine, this script will build a docker image of your model locally. -* wrap-model-in-minikube: If you are using minikube, this script will build a docker image of your model directly on your minikube cluster (for usage see [seldon-core docs](https://github.com/SeldonIO/seldon-core/blob/master/docs/wrappers/readme.md)). - ## Examples The examples in the "models" folder are out-of-the-box machine learning models packaged as required by seldon wrappers. Each model folder usually includes a script to create and save the model, a model python file and a requirements file. @@ -16,4 +10,4 @@ As an example, we describe the content of the folder "models/sklearn_iris". Che * train_iris.py : Script to train and save a sklearn iris classifier * IrisClassifier.py : The file used by seldon-wrappers to load and serve your saved model. * requirements.txt : A list of packages required by your model -* sklearn_iris_deployment.json : A configuration json file used to deploy your model in [seldon-core](https://github.com/SeldonIO/seldon-core#quick-start). \ No newline at end of file +* sklearn_iris_deployment.json : A configuration json file used to deploy your model in [seldon-core](https://github.com/SeldonIO/seldon-core#quick-start). diff --git a/examples/models/sklearn_iris_docker/.dockerignore b/examples/models/sklearn_iris_docker/.dockerignore deleted file mode 100644 index d44c64ac46..0000000000 --- a/examples/models/sklearn_iris_docker/.dockerignore +++ /dev/null @@ -1,4 +0,0 @@ -.dockerignore -Dockerfile -Makefile -sklearn_iris_deployment.json diff --git a/examples/models/sklearn_iris_docker/Dockerfile b/examples/models/sklearn_iris_docker/Dockerfile deleted file mode 100644 index 893f9746c6..0000000000 --- a/examples/models/sklearn_iris_docker/Dockerfile +++ /dev/null @@ -1,84 +0,0 @@ -## Use alpine as build time and runtime image -FROM alpine:3.7 as build-alpine - -## Install build dependencies -RUN apk add --update \ - build-base \ - freetype-dev \ - gcc \ - gfortran \ - libc6-compat \ - libffi-dev \ - libpng-dev \ - openblas-dev \ - openssl-dev \ - py2-pip \ - python2 \ - python2-dev\ - wget \ - && true - -## Symlink missing header, so we can compile numpy -RUN ln -s /usr/include/locale.h /usr/include/xlocale.h - -## Copy package manager config to staging root tree -RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/ -## Install runtime dependencies under staging root tree -RUN apk add --no-cache --initdb --root /out \ - alpine-baselayout \ - busybox \ - ca-certificates \ - freetype \ - libc6-compat \ - libffi \ - libpng \ - libstdc++ \ - musl \ - openblas \ - openssl \ - python2 \ - && true -## Remove package manager residuals -RUN rm -rf /out/etc/apk /out/lib/apk /out/var/cache - -## Enter model source tree and install all Python depenendcies -COPY . /src -WORKDIR /src -## TODO this does take a while to build, maybe a good idea to -## put all related build dependencies into a separate public image -RUN pip install --requirement requirements.txt -## Train the model -RUN python train_iris.py - -## Copy source code and Python dependencies to the saging root tree -RUN mkdir -p /out/src && cp -r /src/* /out/src/ -RUN mkdir -p /out/usr/lib/python2.7/ && cp -r /usr/lib/python2.7/* /out/usr/lib/python2.7/ - -## Use Seldon Core wrapper image to wrap the model source code -FROM seldonio/core-python-wrapper:0.4 as build-wrapper - -ARG MODEL_NAME -ARG IMAGE_VERSION -ARG IMAGE_REPO - -## Copy staging diretory here -COPY --from=build-alpine /out /out -## Wrap the Python model -WORKDIR /wrappers/python -RUN python wrap_model.py /out/src $MODEL_NAME $IMAGE_VERSION $IMAGE_REPO --force - -## Copy wrapped model source code into staging tree and cleanup what is not neccessary at runtime -RUN mkdir -p /out/microservice && cp -r /out/src/build/* /out/microservice/ && rm -rf /out/src -WORKDIR /out/microservice -RUN rm -f Dockerfile Makefile requirements*.txt build_image.sh push_image.sh -## TODO dockerfile doesn't support build argument interpolation in array notation for ENTRYPOINT & CMD -## to get rid of `/bin/sh` wrapper, it'd help to make $MODEL_NAME an environment variable and let the -## Python script pick it up -RUN printf '#!/bin/sh\nexec python microservice.py %s REST --service-type MODEL --persistence 0' $MODEL_NAME > microservice.sh && chmod +x microservice.sh - -## Copy staging root tree onto an empty image -FROM scratch -COPY --from=build-wrapper /out / -WORKDIR /microservice -EXPOSE 5000 -ENTRYPOINT ["/microservice/microservice.sh"] diff --git a/examples/models/sklearn_iris_docker/IrisClassifier.py b/examples/models/sklearn_iris_docker/IrisClassifier.py deleted file mode 100644 index 6b5e9e3fe2..0000000000 --- a/examples/models/sklearn_iris_docker/IrisClassifier.py +++ /dev/null @@ -1,9 +0,0 @@ -from sklearn.externals import joblib - -class IrisClassifier(object): - - def __init__(self): - self.model = joblib.load('IrisClassifier.sav') - - def predict(self,X,features_names): - return self.model.predict_proba(X) diff --git a/examples/models/sklearn_iris_docker/Makefile b/examples/models/sklearn_iris_docker/Makefile deleted file mode 100644 index c3d0f4236e..0000000000 --- a/examples/models/sklearn_iris_docker/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -IMAGE_REPO?=seldonio -IMAGE_NAME?=irisclassifier -IMAGE_VERSION?=0.1 -MODEL_NAME?=IrisClassifier - -container_image: - docker build \ - --build-arg IMAGE_REPO=$(IMAGE_REPO) \ - --build-arg IMAGE_VERSION=$(IMAGE_VERSION) \ - --build-arg MODEL_NAME=$(MODEL_NAME) \ - --tag $(IMAGE_REPO)/$(IMAGE_NAME):$(IMAGE_VERSION) . diff --git a/examples/models/sklearn_iris_docker/requirements.txt b/examples/models/sklearn_iris_docker/requirements.txt deleted file mode 100644 index f02ba9948a..0000000000 --- a/examples/models/sklearn_iris_docker/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -numpy==1.11.2 -pandas==0.18.1 -grpc==0.3.post19 -grpcio==1.8.4 -Flask==0.11.1 -futures -redis==2.10.5 -scipy==0.18.1 -scikit-learn==0.19.0 diff --git a/examples/models/sklearn_iris_docker/sklearn_iris_deployment.json b/examples/models/sklearn_iris_docker/sklearn_iris_deployment.json deleted file mode 100644 index df3204d1dd..0000000000 --- a/examples/models/sklearn_iris_docker/sklearn_iris_deployment.json +++ /dev/null @@ -1,53 +0,0 @@ -{ - "apiVersion": "machinelearning.seldon.io/v1alpha2", - "kind": "SeldonDeployment", - "metadata": { - "labels": { - "app": "seldon" - }, - "name": "seldon-deployment-example" - }, - "spec": { - "annotations": { - "project_name": "Iris classification", - "deployment_version": "0.1" - }, - "name": "sklearn-iris-deployment", - "oauth_key": "oauth-key", - "oauth_secret": "oauth-secret", - "predictors": [ - { - "componentSpecs": [{ - "spec": { - "containers": [ - { - "image": "seldonio/irisclassifier:0.1", - "imagePullPolicy": "IfNotPresent", - "name": "sklearn-iris-classifier", - "resources": { - "requests": { - "memory": "1Mi" - } - } - } - ], - "terminationGracePeriodSeconds": 20 - } - }], - "graph": { - "children": [], - "name": "sklearn-iris-classifier", - "endpoint": { - "type" : "REST" - }, - "type": "MODEL" - }, - "name": "sklearn-iris-predictor", - "replicas": 1, - "annotations": { - "predictor_version" : "0.1" - } - } - ] - } -} diff --git a/examples/models/sklearn_iris_docker/train_iris.py b/examples/models/sklearn_iris_docker/train_iris.py deleted file mode 100644 index 8403db8bf6..0000000000 --- a/examples/models/sklearn_iris_docker/train_iris.py +++ /dev/null @@ -1,25 +0,0 @@ -import numpy as np -import os -from sklearn.linear_model import LogisticRegression -from sklearn.pipeline import Pipeline -from sklearn.externals import joblib -from sklearn import datasets - -def main(): - clf = LogisticRegression() - p = Pipeline([('clf', clf)]) - print 'Training model...' - p.fit(X, y) - print 'Model trained!' - - filename_p = 'IrisClassifier.sav' - print 'Saving model in %s' % filename_p - joblib.dump(p, filename_p) - print 'Model saved!' - -if __name__ == "__main__": - print 'Loading iris data set...' - iris = datasets.load_iris() - X, y = iris.data, iris.target - print 'Dataset loaded!' - main() diff --git a/examples/wrap-model-in-host b/examples/wrap-model-in-host deleted file mode 100755 index 6ee99d5418..0000000000 --- a/examples/wrap-model-in-host +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env bash - -set -o nounset -set -o errexit -set -o pipefail - -MODEL_DIR=$1 -shift -WRAP_MODEL_PARAMS="$@" - -cd $MODEL_DIR -BASE_LOCAL_DIR=$(pwd) - -BASE_VM_DIR=${BASE_LOCAL_DIR} - -set -x -unset DOCKER_TLS_VERIFY -unset DOCKER_HOST -unset DOCKER_CERT_PATH -unset DOCKER_API_VERSION -docker run --rm -it \ - -v ${BASE_VM_DIR}:/work seldonio/core-python-wrapper:0.3 \ - bash -c "rm -rfv /work/build && cd /wrappers/python && python wrap_model.py /work $WRAP_MODEL_PARAMS && ls -1 /work/build" -set +x -cd build && make build_docker_image - diff --git a/examples/wrap-model-in-minikube b/examples/wrap-model-in-minikube deleted file mode 100755 index 5d1ab43a0e..0000000000 --- a/examples/wrap-model-in-minikube +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env bash - -set -o nounset -set -o errexit -set -o pipefail - -MODEL_DIR=$1 -shift -WRAP_MODEL_PARAMS="$@" - -cd $MODEL_DIR -BASE_LOCAL_DIR=$(pwd) - -UNAME_S=$(uname -s) - -BASE_VM_DIR=UNKOWN -if [ ${UNAME_S} = "Darwin" ]; then - BASE_VM_DIR=${BASE_LOCAL_DIR} -fi -if [ ${UNAME_S} = "Linux" ]; then - BASE_VM_DIR=$(echo ${BASE_LOCAL_DIR}|sed -e 's|^/home/|/hosthome/|') -fi - -set -x -eval $(minikube docker-env) -docker run --rm -it \ - -v ${BASE_VM_DIR}:/work seldonio/core-python-wrapper:0.3 \ - bash -c "rm -rfv /work/build && cd /wrappers/python && python wrap_model.py /work $WRAP_MODEL_PARAMS && ls -1 /work/build" -set +x -cd build && make build_docker_image - diff --git a/python/LICENSE b/python/LICENSE new file mode 100644 index 0000000000..9954b4c9df --- /dev/null +++ b/python/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2017 Seldon Technologies Ltd. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/python/MANIFEST.in b/python/MANIFEST.in new file mode 100644 index 0000000000..3455d20e66 --- /dev/null +++ b/python/MANIFEST.in @@ -0,0 +1,3 @@ +include seldon_core/fbs/*.fbs +include seldon_core/proto/*.proto +include seldon_core/openapi/seldon.json diff --git a/python/Makefile b/python/Makefile new file mode 100644 index 0000000000..12a8d220a5 --- /dev/null +++ b/python/Makefile @@ -0,0 +1,83 @@ +SELDON_CORE_DIR=.. +VERSION=0.2.5 + +.PHONY: get_apis +get_apis: + # Protobuf + cp ${SELDON_CORE_DIR}/proto/prediction.proto seldon_core/proto/ + $(MAKE) -C ../proto/tensorflow/ create_protos + cp -r $(SELDON_CORE_DIR)/proto/tensorflow/tensorflow seldon_core/ + $(MAKE) -C ../proto/tensorflow clean + + # Flatbuffers + cp $(SELDON_CORE_DIR)/fbs/prediction.fbs seldon_core/fbs/ + + # OpenAPI + cp $(SELDON_CORE_DIR)/openapi/wrapper.oas3.json seldon_core/openapi/seldon.json + +.PHONY: build_apis +build_apis: + # Protobuf + cd seldon_core && python -m grpc.tools.protoc -I./ --python_out=./ --grpc_python_out=./ ./proto/prediction.proto + sed -i "s/from proto/from seldon_core.proto/g" seldon_core/proto/prediction_pb2_grpc.py + + # Flatbuffers + flatc --python -o seldon_core/fbs seldon_core/fbs/prediction.fbs + +.PHONY: update_version +update_version: + sed -i "s/__version__ = .*/__version__ = '$(VERSION)'/g" seldon_core/__init__.py + +.PHONY: update_package +update_package: get_apis build_apis update_version + +.PHONY: install +install: + pip install -e . + +.PHONY: install-dev +install-dev: + pip install -e . -r requirements.txt + +.PHONY: uninstall +uninstall: + pip uninstall seldon-core + +.PHONY: test +test: + python setup.py test + +.PHONY: build_pypi +build_pypi: + python setup.py sdist bdist_wheel + +.PHONY: build_conda +build_conda: + mkdir -p conda-bld + conda build conda.recipe/meta.yaml -c conda-forge --output-folder conda-bld + +.PHONY: push_pypi_test +push_pypi_test: + twine upload --repository-url https://test.pypi.org/legacy/ dist/* + +.PHONY: push_pypi +push_pypi: + twine upload dist/* + +.PHONY: push_conda +push_conda: + @echo "1st push: need to submit a pull-request to https://github.com/conda-forge/staged-recipes" + @echo "Subsequent pushes: need to for the python-seldon-core feedstock" + @echo "Alternatively use anaconda upload to publish on own channel" + +.PHONY: clean +clean: + rm -rf seldon_core.egg-info + rm -rf seldon_core/tensorflow + rm -rf .eggs + rm -rf .pytest_cache + rm -rf dist + rm -rf conda-bld + rm -rf conda_pkg_path.txt + rm -rf .empty + rm -rf build diff --git a/python/conda.recipe/meta.yaml b/python/conda.recipe/meta.yaml new file mode 100644 index 0000000000..c31be11d15 --- /dev/null +++ b/python/conda.recipe/meta.yaml @@ -0,0 +1,56 @@ +{% set data = load_setup_py_data() %} + +package: + name: python-seldon-core + version: {{ data.version }} + +source: + path: ../ + +build: + noarch: python + number: 0 + script: "{{ PYTHON }} -m pip install --no-deps . -vv" + entry_points: + - seldon-core-microservice = seldon_core.microservice:main + - seldon-core-tester = seldon_core.tester:main + - seldon-core-api-tester = seldon_core.api_tester:main + +requirements: + host: + - python + - pip + - setuptools + run: + - python + - grpcio + - protobuf + - flask + - flask-cors + - redis-py + - tornado + - requests + - numpy + - python-flatbuffers + - tensorflow + +test: + imports: + - seldon_core + requires: + - pytest + source_files: + - tests/* + commands: + - seldon-core-microservice --help + - seldon-core-tester --help + - seldon-core-api-tester --help + - py.test tests + +about: + home: https://github.com/SeldonIO/seldon-core + dev_url: https://github.com/SeldonIO/seldon-core + license: Apache 2.0 + license_family: Apache + license_file: LICENSE + summary: Seldon Core client and microservice wrapper diff --git a/python/readme.rst b/python/readme.rst new file mode 100644 index 0000000000..22fa0e6a33 --- /dev/null +++ b/python/readme.rst @@ -0,0 +1,4 @@ +seldon-core +=========== + +The Python interface to the Seldon Core platform for machine learning deployment. diff --git a/python/requirements.txt b/python/requirements.txt new file mode 100644 index 0000000000..8f2da7b913 --- /dev/null +++ b/python/requirements.txt @@ -0,0 +1,10 @@ +numpy==1.14.5 +pandas==0.23.4 +grpcio==1.14.0 +Flask==1.0.2 +flask-cors==3.0.3 +futures +redis==2.10.5 +tornado==4.5.3 +flatbuffers==1.10 +tensorflow==1.10.1 diff --git a/python/seldon_core/__init__.py b/python/seldon_core/__init__.py new file mode 100644 index 0000000000..a07620f979 --- /dev/null +++ b/python/seldon_core/__init__.py @@ -0,0 +1 @@ +__version__ = '0.2.5_SNAPSHOT' diff --git a/python/seldon_core/api_tester.py b/python/seldon_core/api_tester.py new file mode 100644 index 0000000000..25ad54978b --- /dev/null +++ b/python/seldon_core/api_tester.py @@ -0,0 +1,336 @@ +import argparse +import requests +from requests.auth import HTTPBasicAuth +import numpy as np +import json +import requests +import urllib +from google.protobuf import json_format +from google.protobuf.struct_pb2 import ListValue +import grpc +import sys + +from seldon_core.proto import prediction_pb2 +from seldon_core.proto import prediction_pb2_grpc + + +def array_to_list_value(array, lv=None): + if lv is None: + lv = ListValue() + if len(array.shape) == 1: + lv.extend(array) + else: + for sub_array in array: + sub_lv = lv.add_list() + array_to_list_value(sub_array, sub_lv) + return lv + + +def gen_continuous(range, n): + if range[0] == "inf" and range[1] == "inf": + return np.random.normal(size=n) + if range[0] == "inf": + return range[1] - np.random.lognormal(size=n) + if range[1] == "inf": + return range[0] + np.random.lognormal(size=n) + return np.random.uniform(range[0], range[1], size=n) + + +def reconciliate_cont_type(feature, dtype): + if dtype == "FLOAT": + return feature + if dtype == "INT": + return (feature + 0.5).astype(int).astype(float) + + +def gen_categorical(values, n): + vals = np.random.randint(len(values), size=n) + return np.array(values)[vals].astype(float) + + +def generate_batch(contract, n, field): + feature_batches = [] + for feature_def in contract[field]: + if feature_def["ftype"] == "continuous": + if "range" in feature_def: + range = feature_def["range"] + else: + range = ["inf", "inf"] + if "shape" in feature_def: + shape = [n] + feature_def["shape"] + else: + shape = [n, 1] + batch = gen_continuous(range, shape) + batch = reconciliate_cont_type(batch, feature_def["dtype"]) + elif feature_def["ftype"] == "categorical": + batch = gen_categorical(feature_def["values"], [n, 1]) + feature_batches.append(batch) + return np.concatenate(feature_batches, axis=1) + + +def gen_REST_request(batch, features, tensor=True): + if tensor: + datadef = { + "names": features, + "tensor": { + "shape": batch.shape, + "values": batch.ravel().tolist() + } + } + else: + datadef = { + "names": features, + "ndarray": batch.tolist() + } + + request = { + "meta": {}, + "data": datadef + } + + return request + + +def gen_GRPC_request(batch, features, tensor=True): + if tensor: + datadef = prediction_pb2.DefaultData( + names=features, + tensor=prediction_pb2.Tensor( + shape=batch.shape, + values=batch.ravel().tolist() + ) + ) + else: + datadef = prediction_pb2.DefaultData( + names=features, + ndarray=array_to_list_value(batch) + ) + request = prediction_pb2.SeldonMessage( + data=datadef + ) + return request + + +def unfold_contract(contract): + unfolded_contract = {} + unfolded_contract["targets"] = [] + unfolded_contract["features"] = [] + + for feature in contract["features"]: + if feature.get("repeat") is not None: + for i in range(feature.get("repeat")): + new_feature = {} + new_feature.update(feature) + new_feature["name"] = feature["name"] + str(i + 1) + del new_feature["repeat"] + unfolded_contract["features"].append(new_feature) + else: + unfolded_contract["features"].append(feature) + + for target in contract["targets"]: + if target.get("repeat") is not None: + for i in range(target.get("repeat")): + new_target = {} + new_target.update(target) + new_target["name"] = target["name"] + str(i + 1) + del new_target["repeat"] + unfolded_contract["targets"].append(new_target) + else: + unfolded_contract["targets"].append(target) + + return unfolded_contract + + +def get_token(args): + payload = {'grant_type': 'client_credentials'} + if "oauth_port" in args and not args.oauth_port is None: + port = args.oauth_port + else: + port = args.port + url = "http://" + args.host + ":" + str(port) + "/oauth/token" + print("Getting token from " + url) + response = requests.post( + url, + auth=HTTPBasicAuth(args.oauth_key, args.oauth_secret), + data=payload) + print(response.text) + token = response.json()["access_token"] + return token + + +def run_send_feedback(args): + contract = json.load(open(args.contract, 'r')) + contract = unfold_contract(contract) + feature_names = [feature["name"] for feature in contract["features"]] + response_names = [feature["name"] for feature in contract["targets"]] + + REST_url = "http://" + args.host + ":" + str(args.port) + "/send-feedback" + + for i in range(args.n_requests): + batch = generate_batch(contract, args.batch_size, 'features') + response = generate_batch(contract, args.batch_size, 'targets') + if args.prnt: + print('-' * 40) + print("SENDING NEW REQUEST:") + + if not args.grpc: + REST_request = gen_REST_request( + batch, features=feature_names, tensor=args.tensor) + REST_response = gen_REST_request( + response, features=response_names, tensor=args.tensor) + reward = 1.0 + REST_feedback = {"request": REST_request, + "response": REST_response, "reward": reward} + if args.prnt: + print(REST_feedback) + + if args.oauth_key: + token = get_token(args) + headers = {'Authorization': 'Bearer ' + token} + response = requests.post( + "http://" + args.host + ":" + + str(args.port) + "/api/v0.1/feedback", + json=REST_feedback, + headers=headers + ) + else: + response = requests.post( + "http://" + args.host + ":" + + str(args.port) + args.ambassador_path + + "/api/v0.1/feedback", + json=REST_feedback, + headers=headers + ) + + if args.prnt: + print(response) + + elif args.grpc: + GRPC_request = gen_GRPC_request( + batch, features=feature_names, tensor=args.tensor) + GRPC_response = gen_GRPC_request( + response, features=response_names, tensor=args.tensor) + reward = 1.0 + GRPC_feedback = prediction_pb2.Feedback( + request=GRPC_request, + response=GRPC_response, + reward=reward + ) + + if args.prnt: + print(GRPC_feedback) + + channel = grpc.insecure_channel( + '{}:{}'.format(args.host, args.port)) + stub = prediction_pb2_grpc.SeldonStub(channel) + + if args.oauth_key: + token = get_token(args) + metadata = [('oauth_token', token)] + response = stub.SendFeedback( + request=GRPC_feedback, metadata=metadata) + else: + response = stub.SendFeedback(request=GRPC_feedback) + + if args.prnt: + print("RECEIVED RESPONSE:") + print() + + +def run_predict(args): + contract = json.load(open(args.contract, 'r')) + contract = unfold_contract(contract) + feature_names = [feature["name"] for feature in contract["features"]] + + REST_url = "http://" + args.host + ":" + str(args.port) + "/predict" + + for i in range(args.n_requests): + batch = generate_batch(contract, args.batch_size, 'features') + if args.prnt: + print('-' * 40) + print("SENDING NEW REQUEST:") + + if not args.grpc: + headers = {} + REST_request = gen_REST_request( + batch, features=feature_names, tensor=args.tensor) + if args.prnt: + print(REST_request) + + if args.oauth_key: + token = get_token(args) + headers = {'Authorization': 'Bearer ' + token} + response = requests.post( + "http://" + args.host + ":" + + str(args.port) + "/api/v0.1/predictions", + json=REST_request, + headers=headers + ) + else: + response = requests.post( + "http://" + args.host + ":" + + str(args.port) + args.ambassador_path + + "/api/v0.1/predictions", + json=REST_request, + headers=headers + ) + + jresp = response.json() + + if args.prnt: + print("RECEIVED RESPONSE:") + print(jresp) + print() + else: + GRPC_request = gen_GRPC_request( + batch, features=feature_names, tensor=args.tensor) + if args.prnt: + print(GRPC_request) + + channel = grpc.insecure_channel( + '{}:{}'.format(args.host, args.port)) + stub = prediction_pb2_grpc.SeldonStub(channel) + + if args.oauth_key: + token = get_token(args) + metadata = [('oauth_token', token)] + response = stub.Predict( + request=GRPC_request, metadata=metadata) + else: + response = stub.Predict(request=GRPC_request) + + if args.prnt: + print("RECEIVED RESPONSE:") + print(response) + print() + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("contract", type=str, + help="File that contains the data contract") + parser.add_argument("host", type=str) + parser.add_argument("port", type=int) + parser.add_argument("--endpoint", type=str, + choices=["predict", "send-feedback"], default="predict") + parser.add_argument("-b", "--batch-size", type=int, default=1) + parser.add_argument("-n", "--n-requests", type=int, default=1) + parser.add_argument("--grpc", action="store_true") + parser.add_argument("-t", "--tensor", action="store_true") + parser.add_argument("-p", "--prnt", action="store_true", + help="Prints requests and responses") + parser.add_argument("--oauth-port", type=int) + parser.add_argument("--oauth-key") + parser.add_argument("--oauth-secret") + parser.add_argument("--ambassador-path") + + args = parser.parse_args() + + if args.endpoint == "predict": + run_predict(args) + elif args.endpoint == "send-feedback": + run_send_feedback(args) + + +if __name__ == "__main__": + main() diff --git a/python/seldon_core/fbs/ByteData.py b/python/seldon_core/fbs/ByteData.py new file mode 100644 index 0000000000..c52c21f240 --- /dev/null +++ b/python/seldon_core/fbs/ByteData.py @@ -0,0 +1,46 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class ByteData(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsByteData(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = ByteData() + x.Init(buf, n + offset) + return x + + # ByteData + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # ByteData + def BinData(self, j): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + a = self._tab.Vector(o) + return self._tab.Get(flatbuffers.number_types.Int8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1)) + return 0 + + # ByteData + def BinDataAsNumpy(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Int8Flags, o) + return 0 + + # ByteData + def BinDataLength(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.VectorLen(o) + return 0 + +def ByteDataStart(builder): builder.StartObject(1) +def ByteDataAddBinData(builder, binData): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(binData), 0) +def ByteDataStartBinDataVector(builder, numElems): return builder.StartVector(1, numElems, 1) +def ByteDataEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/Data.py b/python/seldon_core/fbs/Data.py new file mode 100644 index 0000000000..f4d7c65c58 --- /dev/null +++ b/python/seldon_core/fbs/Data.py @@ -0,0 +1,10 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +class Data(object): + NONE = 0 + DefaultData = 1 + ByteData = 2 + StrData = 3 + diff --git a/python/seldon_core/fbs/DefaultData.py b/python/seldon_core/fbs/DefaultData.py new file mode 100644 index 0000000000..27bc0b22ac --- /dev/null +++ b/python/seldon_core/fbs/DefaultData.py @@ -0,0 +1,51 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class DefaultData(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsDefaultData(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = DefaultData() + x.Init(buf, n + offset) + return x + + # DefaultData + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # DefaultData + def Names(self, j): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + a = self._tab.Vector(o) + return self._tab.String(a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4)) + return "" + + # DefaultData + def NamesLength(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.VectorLen(o) + return 0 + + # DefaultData + def Tensor(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + x = self._tab.Indirect(o + self._tab.Pos) + from .Tensor import Tensor + obj = Tensor() + obj.Init(self._tab.Bytes, x) + return obj + return None + +def DefaultDataStart(builder): builder.StartObject(2) +def DefaultDataAddNames(builder, names): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(names), 0) +def DefaultDataStartNamesVector(builder, numElems): return builder.StartVector(4, numElems, 4) +def DefaultDataAddTensor(builder, tensor): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(tensor), 0) +def DefaultDataEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/Meta.py b/python/seldon_core/fbs/Meta.py new file mode 100644 index 0000000000..acc044f03c --- /dev/null +++ b/python/seldon_core/fbs/Meta.py @@ -0,0 +1,54 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class Meta(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsMeta(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = Meta() + x.Init(buf, n + offset) + return x + + # Meta + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # Meta + def Puid(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + + # Meta + def Tags(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + x = self._tab.Indirect(o + self._tab.Pos) + from .TagMap import TagMap + obj = TagMap() + obj.Init(self._tab.Bytes, x) + return obj + return None + + # Meta + def Routing(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8)) + if o != 0: + x = self._tab.Indirect(o + self._tab.Pos) + from .Routing import Routing + obj = Routing() + obj.Init(self._tab.Bytes, x) + return obj + return None + +def MetaStart(builder): builder.StartObject(3) +def MetaAddPuid(builder, puid): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(puid), 0) +def MetaAddTags(builder, tags): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(tags), 0) +def MetaAddRouting(builder, routing): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(routing), 0) +def MetaEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/Routing.py b/python/seldon_core/fbs/Routing.py new file mode 100644 index 0000000000..19393e4179 --- /dev/null +++ b/python/seldon_core/fbs/Routing.py @@ -0,0 +1,38 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class Routing(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsRouting(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = Routing() + x.Init(buf, n + offset) + return x + + # Routing + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # Routing + def Id(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + + # Routing + def Route(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Int32Flags, o + self._tab.Pos) + return 0 + +def RoutingStart(builder): builder.StartObject(2) +def RoutingAddId(builder, id): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(id), 0) +def RoutingAddRoute(builder, route): builder.PrependInt32Slot(1, route, 0) +def RoutingEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/SeldonMessage.py b/python/seldon_core/fbs/SeldonMessage.py new file mode 100644 index 0000000000..4d6ac44ef9 --- /dev/null +++ b/python/seldon_core/fbs/SeldonMessage.py @@ -0,0 +1,73 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class SeldonMessage(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsSeldonMessage(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = SeldonMessage() + x.Init(buf, n + offset) + return x + + # SeldonMessage + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # SeldonMessage + def Protocol(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Int32Flags, o + self._tab.Pos) + return 134361921 + + # SeldonMessage + def Status(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + x = self._tab.Indirect(o + self._tab.Pos) + from .Status import Status + obj = Status() + obj.Init(self._tab.Bytes, x) + return obj + return None + + # SeldonMessage + def Meta(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8)) + if o != 0: + x = self._tab.Indirect(o + self._tab.Pos) + from .Meta import Meta + obj = Meta() + obj.Init(self._tab.Bytes, x) + return obj + return None + + # SeldonMessage + def DataType(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos) + return 0 + + # SeldonMessage + def Data(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(12)) + if o != 0: + from flatbuffers.table import Table + obj = Table(bytearray(), 0) + self._tab.Union(obj, o) + return obj + return None + +def SeldonMessageStart(builder): builder.StartObject(5) +def SeldonMessageAddProtocol(builder, protocol): builder.PrependInt32Slot(0, protocol, 134361921) +def SeldonMessageAddStatus(builder, status): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(status), 0) +def SeldonMessageAddMeta(builder, meta): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(meta), 0) +def SeldonMessageAddDataType(builder, dataType): builder.PrependUint8Slot(3, dataType, 0) +def SeldonMessageAddData(builder, data): builder.PrependUOffsetTRelativeSlot(4, flatbuffers.number_types.UOffsetTFlags.py_type(data), 0) +def SeldonMessageEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/SeldonMethod.py b/python/seldon_core/fbs/SeldonMethod.py new file mode 100644 index 0000000000..2afed73e6f --- /dev/null +++ b/python/seldon_core/fbs/SeldonMethod.py @@ -0,0 +1,8 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +class SeldonMethod(object): + PREDICT = 0 + RESPONSE = 1 + diff --git a/python/seldon_core/fbs/SeldonPayload.py b/python/seldon_core/fbs/SeldonPayload.py new file mode 100644 index 0000000000..9738c644a0 --- /dev/null +++ b/python/seldon_core/fbs/SeldonPayload.py @@ -0,0 +1,8 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +class SeldonPayload(object): + NONE = 0 + SeldonMessage = 1 + diff --git a/python/seldon_core/fbs/SeldonProtocolVersion.py b/python/seldon_core/fbs/SeldonProtocolVersion.py new file mode 100644 index 0000000000..7b91df7cec --- /dev/null +++ b/python/seldon_core/fbs/SeldonProtocolVersion.py @@ -0,0 +1,7 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +class SeldonProtocolVersion(object): + V1 = 134361921 + diff --git a/python/seldon_core/fbs/SeldonRPC.py b/python/seldon_core/fbs/SeldonRPC.py new file mode 100644 index 0000000000..a1644c9f72 --- /dev/null +++ b/python/seldon_core/fbs/SeldonRPC.py @@ -0,0 +1,49 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class SeldonRPC(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsSeldonRPC(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = SeldonRPC() + x.Init(buf, n + offset) + return x + + # SeldonRPC + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # SeldonRPC + def Method(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Int8Flags, o + self._tab.Pos) + return 0 + + # SeldonRPC + def MessageType(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Uint8Flags, o + self._tab.Pos) + return 0 + + # SeldonRPC + def Message(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8)) + if o != 0: + from flatbuffers.table import Table + obj = Table(bytearray(), 0) + self._tab.Union(obj, o) + return obj + return None + +def SeldonRPCStart(builder): builder.StartObject(3) +def SeldonRPCAddMethod(builder, method): builder.PrependInt8Slot(0, method, 0) +def SeldonRPCAddMessageType(builder, messageType): builder.PrependUint8Slot(1, messageType, 0) +def SeldonRPCAddMessage(builder, message): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(message), 0) +def SeldonRPCEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/Status.py b/python/seldon_core/fbs/Status.py new file mode 100644 index 0000000000..36af22dda6 --- /dev/null +++ b/python/seldon_core/fbs/Status.py @@ -0,0 +1,54 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class Status(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsStatus(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = Status() + x.Init(buf, n + offset) + return x + + # Status + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # Status + def Code(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Int32Flags, o + self._tab.Pos) + return 0 + + # Status + def Info(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + + # Status + def Reason(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + + # Status + def Status(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(10)) + if o != 0: + return self._tab.Get(flatbuffers.number_types.Int8Flags, o + self._tab.Pos) + return 0 + +def StatusStart(builder): builder.StartObject(4) +def StatusAddCode(builder, code): builder.PrependInt32Slot(0, code, 0) +def StatusAddInfo(builder, info): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(info), 0) +def StatusAddReason(builder, reason): builder.PrependUOffsetTRelativeSlot(2, flatbuffers.number_types.UOffsetTFlags.py_type(reason), 0) +def StatusAddStatus(builder, status): builder.PrependInt8Slot(3, status, 0) +def StatusEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/StatusValue.py b/python/seldon_core/fbs/StatusValue.py new file mode 100644 index 0000000000..b5dfc7d603 --- /dev/null +++ b/python/seldon_core/fbs/StatusValue.py @@ -0,0 +1,8 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +class StatusValue(object): + SUCCESS = 0 + FAILURE = 1 + diff --git a/python/seldon_core/fbs/StrData.py b/python/seldon_core/fbs/StrData.py new file mode 100644 index 0000000000..61a9bda6bc --- /dev/null +++ b/python/seldon_core/fbs/StrData.py @@ -0,0 +1,30 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class StrData(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsStrData(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = StrData() + x.Init(buf, n + offset) + return x + + # StrData + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # StrData + def StrData(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + +def StrDataStart(builder): builder.StartObject(1) +def StrDataAddStrData(builder, strData): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(strData), 0) +def StrDataEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/TagMap.py b/python/seldon_core/fbs/TagMap.py new file mode 100644 index 0000000000..fba3a35d43 --- /dev/null +++ b/python/seldon_core/fbs/TagMap.py @@ -0,0 +1,38 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class TagMap(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsTagMap(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = TagMap() + x.Init(buf, n + offset) + return x + + # TagMap + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # TagMap + def Key(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + + # TagMap + def Value(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.String(o + self._tab.Pos) + return None + +def TagMapStart(builder): builder.StartObject(2) +def TagMapAddKey(builder, key): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(key), 0) +def TagMapAddValue(builder, value): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(value), 0) +def TagMapEnd(builder): return builder.EndObject() diff --git a/python/seldon_core/fbs/Tensor.py b/python/seldon_core/fbs/Tensor.py new file mode 100644 index 0000000000..ddbeb5bedb --- /dev/null +++ b/python/seldon_core/fbs/Tensor.py @@ -0,0 +1,70 @@ +# automatically generated by the FlatBuffers compiler, do not modify + +# namespace: + +import flatbuffers + +class Tensor(object): + __slots__ = ['_tab'] + + @classmethod + def GetRootAsTensor(cls, buf, offset): + n = flatbuffers.encode.Get(flatbuffers.packer.uoffset, buf, offset) + x = Tensor() + x.Init(buf, n + offset) + return x + + # Tensor + def Init(self, buf, pos): + self._tab = flatbuffers.table.Table(buf, pos) + + # Tensor + def Shape(self, j): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + a = self._tab.Vector(o) + return self._tab.Get(flatbuffers.number_types.Int32Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 4)) + return 0 + + # Tensor + def ShapeAsNumpy(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Int32Flags, o) + return 0 + + # Tensor + def ShapeLength(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4)) + if o != 0: + return self._tab.VectorLen(o) + return 0 + + # Tensor + def Values(self, j): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + a = self._tab.Vector(o) + return self._tab.Get(flatbuffers.number_types.Float64Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 8)) + return 0 + + # Tensor + def ValuesAsNumpy(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.GetVectorAsNumpy(flatbuffers.number_types.Float64Flags, o) + return 0 + + # Tensor + def ValuesLength(self): + o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6)) + if o != 0: + return self._tab.VectorLen(o) + return 0 + +def TensorStart(builder): builder.StartObject(2) +def TensorAddShape(builder, shape): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(shape), 0) +def TensorStartShapeVector(builder, numElems): return builder.StartVector(4, numElems, 4) +def TensorAddValues(builder, values): builder.PrependUOffsetTRelativeSlot(1, flatbuffers.number_types.UOffsetTFlags.py_type(values), 0) +def TensorStartValuesVector(builder, numElems): return builder.StartVector(8, numElems, 8) +def TensorEnd(builder): return builder.EndObject() diff --git a/wrappers/python/__init__.py b/python/seldon_core/fbs/__init__.py similarity index 100% rename from wrappers/python/__init__.py rename to python/seldon_core/fbs/__init__.py diff --git a/python/seldon_core/fbs/prediction.fbs b/python/seldon_core/fbs/prediction.fbs new file mode 100644 index 0000000000..be2779a601 --- /dev/null +++ b/python/seldon_core/fbs/prediction.fbs @@ -0,0 +1,62 @@ +enum StatusValue : byte { SUCCESS = 0, FAILURE = 1 } + +union Data { DefaultData, ByteData, StrData } + +enum SeldonMethod : byte { PREDICT = 0, RESPONSE = 1 } + +enum SeldonProtocolVersion : int32 { V1 = 134361921 } + +union SeldonPayload { SeldonMessage } + +table SeldonRPC { + method:SeldonMethod; + message:SeldonPayload; +} + +table SeldonMessage { + protocol:SeldonProtocolVersion = V1; + status:Status; + meta:Meta; + data:Data; +} + +table ByteData { + binData:[byte]; +} + +table StrData { + strData:string; +} + +table DefaultData { + names:[string]; + tensor:Tensor; +} + +table Tensor { + shape:[int32]; + values:[double]; +} + +table Meta { + puid:string; + tags:TagMap; + routing:Routing; +} + +table TagMap { + key:string; + value:string; +} + +table Routing { + id:string; + route:int32; + } + +table Status { + code:int32; + info:string; + reason:string; + status:StatusValue; +} diff --git a/wrappers/python/metrics.py b/python/seldon_core/metrics.py similarity index 62% rename from wrappers/python/metrics.py rename to python/seldon_core/metrics.py index 7df7c1a68c..5d0fdec524 100644 --- a/wrappers/python/metrics.py +++ b/python/seldon_core/metrics.py @@ -1,21 +1,26 @@ -from microservice import SeldonMicroserviceException import json +from seldon_core.microservice import SeldonMicroserviceException + COUNTER = "COUNTER" GAUGE = "GAUGE" TIMER = "TIMER" -def create_counter(key,value): + +def create_counter(key, value): test = value + 1 - return {"key":key,"type":COUNTER,"value":value} + return {"key": key, "type": COUNTER, "value": value} -def create_gauge(key,value): + +def create_gauge(key, value): test = value + 1 - return {"key":key,"type":GAUGE,"value":value} + return {"key": key, "type": GAUGE, "value": value} + -def create_timer(key,value): +def create_timer(key, value): test = value + 1 - return {"key":key,"type":TIMER,"value":value} + return {"key": key, "type": TIMER, "value": value} + def validate_metrics(metrics): if isinstance(metrics, (list,)): @@ -32,14 +37,14 @@ def validate_metrics(metrics): return False return True + def get_custom_metrics(component): - if hasattr(component,"metrics"): + if hasattr(component, "metrics"): metrics = component.metrics() if not validate_metrics(metrics): jStr = json.dumps(metrics) - raise SeldonMicroserviceException("Bad metric created during request: "+jStr,reason="MICROSERVICE_BAD_METRIC") + raise SeldonMicroserviceException( + "Bad metric created during request: " + jStr, reason="MICROSERVICE_BAD_METRIC") return metrics else: return None - - diff --git a/python/seldon_core/microservice.py b/python/seldon_core/microservice.py new file mode 100644 index 0000000000..5da289a524 --- /dev/null +++ b/python/seldon_core/microservice.py @@ -0,0 +1,357 @@ +from flask import Flask, Blueprint, request +import argparse +import numpy as np +import os +import importlib +import json +import time +import logging +import multiprocessing as mp +import tensorflow as tf +from tensorflow.core.framework.tensor_pb2 import TensorProto +from google.protobuf import json_format +from google.protobuf.struct_pb2 import ListValue +import sys + +from seldon_core.proto import prediction_pb2 +import seldon_core.persistence as persistence + +logger = logging.getLogger(__name__) + +PARAMETERS_ENV_NAME = "PREDICTIVE_UNIT_PARAMETERS" +SERVICE_PORT_ENV_NAME = "PREDICTIVE_UNIT_SERVICE_PORT" +DEFAULT_PORT = 5000 + +DEBUG_PARAMETER = "SELDON_DEBUG" +DEBUG = False + +ANNOTATIONS_FILE = "/etc/podinfo/annotations" + + +def startServers(target1, target2): + p2 = mp.Process(target=target2) + p2.deamon = True + p2.start() + + target1() + + p2.join() + + +class SeldonMicroserviceException(Exception): + status_code = 400 + + def __init__(self, message, status_code=None, payload=None, reason="MICROSERVICE_BAD_DATA"): + Exception.__init__(self) + self.message = message + if status_code is not None: + self.status_code = status_code + self.payload = payload + self.reason = reason + + def to_dict(self): + rv = {"status": {"status": 1, "info": self.message, + "code": -1, "reason": self.reason}} + return rv + + +def sanity_check_request(req): + if not type(req) == dict: + raise SeldonMicroserviceException("Request must be a dictionary") + if "data" in req: + data = req.get("data") + if not type(data) == dict: + raise SeldonMicroserviceException( + "data field must be a dictionary") + if data.get('ndarray') is None and data.get('tensor') is None and data.get('tftensor') is None: + raise SeldonMicroserviceException( + "Data dictionary has no 'tensor', 'ndarray' or 'tftensor' keyword.") + elif not ("binData" in req or "strData" in req): + raise SeldonMicroserviceException("Request must contain Default Data") + # TODO: Should we check more things? Like shape not being None or empty for a tensor? + + +def extract_message(): + jStr = request.form.get("json") + if jStr: + message = json.loads(jStr) + else: + jStr = request.args.get('json') + if jStr: + message = json.loads(jStr) + else: + raise SeldonMicroserviceException("Empty json parameter in data") + if message is None: + raise SeldonMicroserviceException("Invalid Data Format") + return message + + +def get_custom_tags(component): + if hasattr(component, "tags"): + return component.tags() + else: + return None + + +def array_to_list_value(array, lv=None): + if lv is None: + lv = ListValue() + if len(array.shape) == 1: + lv.extend(array) + else: + for sub_array in array: + sub_lv = lv.add_list() + array_to_list_value(sub_array, sub_lv) + return lv + + +def get_data_from_json(message): + if "data" in message: + datadef = message.get("data") + return rest_datadef_to_array(datadef) + elif "binData" in message: + return message["binData"] + elif "strData" in message: + return message["strData"] + else: + strJson = json.dumps(message) + raise SeldonMicroserviceException( + "Can't find data in json: " + strJson) + + +def rest_datadef_to_array(datadef): + if datadef.get("tensor") is not None: + features = np.array(datadef.get("tensor").get("values")).reshape( + datadef.get("tensor").get("shape")) + elif datadef.get("ndarray") is not None: + features = np.array(datadef.get("ndarray")) + elif datadef.get("tftensor") is not None: + tfp = TensorProto() + json_format.ParseDict(datadef.get("tftensor"), + tfp, ignore_unknown_fields=False) + features = tf.make_ndarray(tfp) + else: + features = np.array([]) + return features + + +def array_to_rest_datadef(array, names, original_datadef): + datadef = {"names": names} + if original_datadef.get("tensor") is not None: + datadef["tensor"] = { + "shape": array.shape, + "values": array.ravel().tolist() + } + elif original_datadef.get("ndarray") is not None: + datadef["ndarray"] = array.tolist() + elif original_datadef.get("tftensor") is not None: + tftensor = tf.make_tensor_proto(array) + jStrTensor = json_format.MessageToJson(tftensor) + jTensor = json.loads(jStrTensor) + datadef["tftensor"] = jTensor + else: + datadef["ndarray"] = array.tolist() + return datadef + + +def get_data_from_proto(request): + data_type = request.WhichOneof("data_oneof") + if data_type == "data": + datadef = request.data + return grpc_datadef_to_array(datadef) + elif data_type == "binData": + return request.binData + elif data_type == "strData": + return request.strData + else: + raise SeldonMicroserviceException("Unknown data in SeldonMessage") + + +def grpc_datadef_to_array(datadef): + data_type = datadef.WhichOneof("data_oneof") + if data_type == "tensor": + if (sys.version_info >= (3, 0)): + sz = np.prod(datadef.tensor.shape) # get number of float64 entries + c = datadef.tensor.SerializeToString() # get bytes + # create array from packed entries which are at end of bytes - assumes same endianness + features = np.frombuffer(memoryview( + c[-(sz * 8):]), dtype=np.float64, count=sz, offset=0) + features = features.reshape(datadef.tensor.shape) + else: + # Python 2 version which is slower + features = np.array(datadef.tensor.values).reshape( + datadef.tensor.shape) + elif data_type == "ndarray": + features = np.array(datadef.ndarray) + elif data_type == "tftensor": + features = tf.make_ndarray(datadef.tftensor) + else: + features = np.array([]) + return features + + +def array_to_grpc_datadef(array, names, data_type): + if data_type == "tensor": + datadef = prediction_pb2.DefaultData( + names=names, + tensor=prediction_pb2.Tensor( + shape=array.shape, + values=array.ravel().tolist() + ) + ) + elif data_type == "ndarray": + datadef = prediction_pb2.DefaultData( + names=names, + ndarray=array_to_list_value(array) + ) + elif data_type == "tftensor": + datadef = prediction_pb2.DefaultData( + names=names, + tftensor=tf.make_tensor_proto(array) + ) + else: + datadef = prediction_pb2.DefaultData( + names=names, + ndarray=array_to_list_value(array) + ) + + return datadef + + +def parse_parameters(parameters): + type_dict = { + "INT": int, + "FLOAT": float, + "DOUBLE": float, + "STRING": str, + "BOOL": bool + } + parsed_parameters = {} + for param in parameters: + name = param.get("name") + value = param.get("value") + type_ = param.get("type") + parsed_parameters[name] = type_dict[type_](value) + return parsed_parameters + + +def load_annotations(): + annotations = {} + try: + if os.path.isfile(ANNOTATIONS_FILE): + with open(ANNOTATIONS_FILE, "r") as ins: + for line in ins: + line = line.rstrip() + parts = line.split("=") + if len(parts) == 2: + value = parts[1] + value = parts[1][1:-1] + logger.info("Found annotation %s:%s ", parts[0], value) + annotations[parts[0]] = value + else: + logger.info("bad annotation [%s]", line) + except: + logger.error("Failed to open annotations file %s", ANNOTATIONS_FILE) + return annotations + + +def main(): + LOG_FORMAT = '%(asctime)s - %(name)s:%(funcName)s:%(lineno)s - %(levelname)s: %(message)s' + logging.basicConfig(level=logging.INFO, format=LOG_FORMAT) + logger.info('Starting microservice.py:main') + + sys.path.append(os.getcwd()) + parser = argparse.ArgumentParser() + parser.add_argument("interface_name", type=str, + help="Name of the user interface.") + parser.add_argument("api_type", type=str, choices=["REST", "GRPC", "FBS"]) + + parser.add_argument("--service-type", type=str, choices=[ + "MODEL", "ROUTER", "TRANSFORMER", "COMBINER", "OUTLIER_DETECTOR"], default="MODEL") + parser.add_argument("--persistence", nargs='?', + default=0, const=1, type=int) + parser.add_argument("--parameters", type=str, + default=os.environ.get(PARAMETERS_ENV_NAME, "[]")) + parser.add_argument("--log-level", type=str, default='INFO') + args = parser.parse_args() + + parameters = parse_parameters(json.loads(args.parameters)) + + # set up log level + log_level_num = getattr(logging, args.log_level.upper(), None) + if not isinstance(log_level_num, int): + raise ValueError('Invalid log level: %s', args.log_level) + logger.setLevel(log_level_num) + + DEBUG = False + if parameters.get(DEBUG_PARAMETER): + parameters.pop(DEBUG_PARAMETER) + DEBUG = True + + annotations = load_annotations() + logger.info("Annotations: %s", annotations) + + interface_file = importlib.import_module(args.interface_name) + user_class = getattr(interface_file, args.interface_name) + + if args.persistence: + logger.info('Restoring persisted component') + user_object = persistence.restore(user_class, parameters, debug=DEBUG) + persistence.persist(user_object, parameters.get("push_frequency")) + else: + user_object = user_class(**parameters) + + if args.service_type == "MODEL": + import seldon_core.model_microservice as seldon_microservice + elif args.service_type == "ROUTER": + import seldon_core.router_microservice as seldon_microservice + elif args.service_type == "TRANSFORMER": + import seldon_core.transformer_microservice as seldon_microservice + elif args.service_type == "OUTLIER_DETECTOR": + import seldon_core.outlier_detector_microservice as seldon_microservice + + port = int(os.environ.get(SERVICE_PORT_ENV_NAME, DEFAULT_PORT)) + + if args.api_type == "REST": + def rest_prediction_server(): + app = seldon_microservice.get_rest_microservice( + user_object, debug=DEBUG) + app.run(host='0.0.0.0', port=port) + + logger.info(f"REST microservice running on port {port}") + server1_func = rest_prediction_server + + elif args.api_type == "GRPC": + def grpc_prediction_server(): + server = seldon_microservice.get_grpc_server( + user_object, debug=DEBUG, annotations=annotations) + server.add_insecure_port("0.0.0.0:{}".format(port)) + server.start() + + logger.info(f"GRPC microservice Running on port {port}") + while True: + time.sleep(1000) + + server1_func = grpc_prediction_server + + elif args.api_type == "FBS": + def fbs_prediction_server(): + seldon_microservice.run_flatbuffers_server(user_object, port) + + logger.info(f"FBS microservice Running on port {port}") + server1_func = fbs_prediction_server + + else: + server1_func = None + + if hasattr(user_object, 'custom_service') and callable(getattr(user_object, 'custom_service')): + server2_func = user_object.custom_service + else: + server2_func = None + + logger.info('Starting servers') + startServers(server1_func, server2_func) + + +if __name__ == "__main__": + main() diff --git a/python/seldon_core/model_microservice.py b/python/seldon_core/model_microservice.py new file mode 100644 index 0000000000..9b6a416518 --- /dev/null +++ b/python/seldon_core/model_microservice.py @@ -0,0 +1,250 @@ +import grpc +from concurrent import futures +from google.protobuf import json_format + +from flask import jsonify, Flask, send_from_directory +from flask_cors import CORS +import numpy as np +import logging + +from tornado.tcpserver import TCPServer +from tornado.iostream import StreamClosedError +from tornado import gen +import tornado.ioloop +import struct +import traceback +import os + +from seldon_core.proto import prediction_pb2, prediction_pb2_grpc +from seldon_core.microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ + array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ + SeldonMicroserviceException, get_custom_tags, get_data_from_json, get_data_from_proto +from seldon_core.metrics import get_custom_metrics +from seldon_core.seldon_flatbuffers import SeldonRPCToNumpyArray, NumpyArrayToSeldonRPC, CreateErrorMsg + +PRED_UNIT_ID = os.environ.get("PREDICTIVE_UNIT_ID") + +logger = logging.getLogger(__name__) + +# --------------------------- +# Interaction with user model +# --------------------------- + + +def predict(user_model, features, feature_names): + return user_model.predict(features, feature_names) + + +def send_feedback(user_model, features, feature_names, reward, truth): + if hasattr(user_model, "send_feedback"): + user_model.send_feedback(features, feature_names, reward, truth) + + +def get_class_names(user_model, n_targets): + if hasattr(user_model, "class_names"): + return user_model.class_names + else: + return ["t:{}".format(i) for i in range(n_targets)] + + +# ---------------------------- +# REST +# ---------------------------- + +def get_rest_microservice(user_model, debug=False): + + app = Flask(__name__, static_url_path='') + CORS(app) + + @app.errorhandler(SeldonMicroserviceException) + def handle_invalid_usage(error): + response = jsonify(error.to_dict()) + logger.error("%s", error.to_dict()) + response.status_code = 400 + return response + + @app.route("/seldon.json", methods=["GET"]) + def openAPI(): + return send_from_directory('', "seldon.json") + + @app.route("/predict", methods=["GET", "POST"]) + def Predict(): + request = extract_message() + logger.debug("Request: %s", request) + + sanity_check_request(request) + + features = get_data_from_json(request) + names = request.get("data", {}).get("names") + + predictions = predict(user_model, features, names) + logger.debug("Predictions: %s", predictions) + + # If predictions is an numpy array or we used the default data then return as numpy array + if isinstance(predictions, np.ndarray) or "data" in request: + predictions = np.array(predictions) + if len(predictions.shape) > 1: + class_names = get_class_names(user_model, predictions.shape[1]) + else: + class_names = [] + data = array_to_rest_datadef( + predictions, class_names, request.get("data", {})) + response = {"data": data, "meta": {}} + else: + response = {"binData": predictions, "meta": {}} + + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) + + @app.route("/send-feedback", methods=["GET", "POST"]) + def SendFeedback(): + feedback = extract_message() + logger.debug("Feedback received: %s", feedback) + + datadef_request = feedback.get("request", {}).get("data", {}) + features = rest_datadef_to_array(datadef_request) + + datadef_truth = feedback.get("truth", {}).get("data", {}) + truth = rest_datadef_to_array(datadef_truth) + + reward = feedback.get("reward") + + send_feedback(user_model, features, + datadef_request.get("names"), reward, truth) + return jsonify({}) + + return app + + +# ---------------------------- +# GRPC +# ---------------------------- + +class SeldonModelGRPC(object): + def __init__(self, user_model): + self.user_model = user_model + + def Predict(self, request, context): + features = get_data_from_proto(request) + datadef = request.data + data_type = request.WhichOneof("data_oneof") + predictions = predict(self.user_model, features, datadef.names) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson, meta) + + if isinstance(predictions, np.ndarray) or data_type == "data": + predictions = np.array(predictions) + if len(predictions.shape) > 1: + class_names = get_class_names( + self.user_model, predictions.shape[1]) + else: + class_names = [] + + if data_type == "data": + default_data_type = request.data.WhichOneof("data_oneof") + else: + default_data_type = "tensor" + data = array_to_grpc_datadef( + predictions, class_names, default_data_type) + return prediction_pb2.SeldonMessage(data=data, meta=meta) + else: + return prediction_pb2.SeldonMessage(binData=predictions, meta=meta) + + def SendFeedback(self, feedback, context): + datadef_request = feedback.request.data + features = grpc_datadef_to_array(datadef_request) + + truth = grpc_datadef_to_array(feedback.truth) + reward = feedback.reward + + send_feedback(self.user_model, features, + datadef_request.names, truth, reward) + + return prediction_pb2.SeldonMessage() + + +ANNOTATION_GRPC_MAX_MSG_SIZE = 'seldon.io/grpc-max-message-size' + + +def get_grpc_server(user_model, debug=False, annotations={}): + seldon_model = SeldonModelGRPC(user_model) + options = [] + if ANNOTATION_GRPC_MAX_MSG_SIZE in annotations: + max_msg = int(annotations[ANNOTATION_GRPC_MAX_MSG_SIZE]) + logger.info( + "Setting grpc max message and receive length to %d", max_msg) + options.append(('grpc.max_message_length', max_msg)) + options.append(('grpc.max_receive_message_length', max_msg)) + + server = grpc.server(futures.ThreadPoolExecutor( + max_workers=10), options=options) + prediction_pb2_grpc.add_ModelServicer_to_server(seldon_model, server) + + return server + + +# ---------------------------- +# Flatbuffers (experimental) +# ---------------------------- + +class SeldonFlatbuffersServer(TCPServer): + def __init__(self, user_model): + super(SeldonFlatbuffersServer, self).__init__() + self.user_model = user_model + + @gen.coroutine + def handle_stream(self, stream, address): + while True: + try: + data = yield stream.read_bytes(4) + obj = struct.unpack(' 1: + class_names = get_class_names( + self.user_model, predictions.shape[1]) + else: + class_names = [] + outData = NumpyArrayToSeldonRPC(predictions, class_names) + yield stream.write(outData) + except StreamClosedError: + logger.exception( + "Stream closed during processing:", address) + break + except Exception: + tb = traceback.format_exc() + logger.exception( + "Caught exception during processing:", address, tb) + outData = CreateErrorMsg(tb) + yield stream.write(outData) + stream.close() + break + except StreamClosedError: + logger.exception( + "Stream closed during data inputstream read:", address) + break + + +def run_flatbuffers_server(user_model, port, debug=False): + server = SeldonFlatbuffersServer(user_model) + server.listen(port) + logger.info("Tornado Server listening on port %s", port) + tornado.ioloop.IOLoop.current().start() diff --git a/python/seldon_core/openapi/seldon.json b/python/seldon_core/openapi/seldon.json new file mode 100644 index 0000000000..e850a4d962 --- /dev/null +++ b/python/seldon_core/openapi/seldon.json @@ -0,0 +1,899 @@ +{ + "openapi": "3.0.0", + "info": { + "title": "Seldon External API", + "version": "0.1", + "contact": { + "name": "Seldon Core", + "url": "https://github.com/SeldonIO/seldon-core" + } + }, + "externalDocs": { + "description": "Seldon Core Documentation", + "url": "https://github.com/SeldonIO/seldon-core" + }, + "servers": [ + { + "url": "http://{host}:{port}", + "variables": { + "host": { + "default": "localhost", + "description": "host running seldon core" + }, + "port": { + "default": "80" + } + } + }, + { + "url": "http://localhost:8002", + "description": "fixed host as swagger UI has bug with variables for auth" + } + ], + "paths": { + "/aggregate": { + "post": { + "operationId": "Aggregate2", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/SeldonMessageList" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + }, + "get": { + "operationId": "Aggregate", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "body", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/SeldonMessageList" + } + } + ], + "tags": [ + "Internal" + ] + } + }, + "/predict": { + "get": { + "operationId": "TransformInput4", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "json", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + }, + "example": { + "json": "{\"data\":{\"names\" : [\"feature1\"],\"tensor\" : {\"shape\": [1,1],\"values\": [1]}}}" + } + } + ], + "tags": [ + "Internal" + ] + }, + "post": { + "operationId": "TransformInput3", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + } + }, + "/route": { + "get": { + "operationId": "Route2", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "json", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + ], + "tags": [ + "Internal" + ] + }, + "post": { + "operationId": "Route", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + } + }, + "/send-feedback": { + "get": { + "operationId": "SendFeedback2", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "json", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/Feedback" + } + } + ], + "tags": [ + "Internal" + ] + }, + "post": { + "operationId": "SendFeedback", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/Feedback" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + } + }, + "/transform-input": { + "get": { + "operationId": "TransformInput2", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "json", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + ], + "tags": [ + "Internal" + ] + }, + "post": { + "operationId": "TransformInput", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + } + }, + "/transform-output": { + "get": { + "operationId": "TransformOutput2", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "parameters": [ + { + "name": "json", + "in": "query", + "required": true, + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + ], + "tags": [ + "Internal" + ] + }, + "post": { + "operationId": "TransformOutput", + "responses": { + "200": { + "description": "A successful response.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + } + }, + "tags": [ + "Internal" + ], + "requestBody": { + "content": { + "application/x-www-form-urlencoded": { + "schema": { + "type": "object", + "properties": { + "json": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + }, + "encoding": { + "json": { + "contentType": "application/json" + } + } + } + }, + "required": true + } + } + } + }, + "components": { + "schemas": { + "StatusStatusFlag": { + "type": "string", + "enum": [ + "SUCCESS", + "FAILURE" + ], + "default": "SUCCESS" + }, + "AnyValue": { + "description": "Can be anything: string, number, array, object, etc." + }, + "MetricType": { + "type": "string", + "enum": [ + "COUNTER", + "GAUGE", + "TIMER" + ], + "default": "COUNTER" + }, + "Metric": { + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/MetricType" + }, + "key": { + "type": "string" + }, + "value": { + "type": "number", + "format": "float" + } + } + }, + "DefaultData": { + "type": "object", + "properties": { + "names": { + "type": "array", + "items": { + "type": "string" + } + }, + "tensor": { + "$ref": "#/components/schemas/Tensor" + }, + "ndarry": { + "type": "array", + "items": { + "$ref": "#/components/schemas/AnyValue" + } + }, + "tftensor": { + "$ref": "#/components/schemas/TensorflowTensorProto" + } + } + }, + "Feedback": { + "type": "object", + "properties": { + "request": { + "$ref": "#/components/schemas/SeldonMessage" + }, + "response": { + "$ref": "#/components/schemas/SeldonMessage" + }, + "reward": { + "type": "number", + "format": "float" + }, + "truth": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + }, + "Meta": { + "type": "object", + "properties": { + "puid": { + "type": "string" + }, + "tags": { + "type": "object", + "additionalProperties": { + "$ref": "#/components/schemas/AnyValue" + }, + "example": { + "mytag": "myvalue" + } + }, + "routing": { + "type": "object", + "additionalProperties": { + "type": "integer", + "format": "int32" + }, + "example": { + "router1": 1 + } + }, + "requestPath": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "example": { + "classifier": "seldonio/mock_classifier:1.0" + } + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Metric" + } + } + } + }, + "SeldonMessage": { + "type": "object", + "properties": { + "status": { + "$ref": "#/components/schemas/Status" + }, + "meta": { + "$ref": "#/components/schemas/Meta" + }, + "data": { + "$ref": "#/components/schemas/DefaultData" + }, + "binData": { + "type": "string", + "format": "byte" + }, + "strData": { + "type": "string" + } + } + }, + "SeldonMessageList": { + "type": "object", + "properties": { + "seldonMessages": { + "type": "array", + "items": { + "$ref": "#/components/schemas/SeldonMessage" + } + } + } + }, + "Status": { + "type": "object", + "properties": { + "code": { + "type": "integer", + "format": "int32" + }, + "info": { + "type": "string" + }, + "reason": { + "type": "string" + }, + "status": { + "$ref": "#/components/schemas/StatusStatusFlag" + } + } + }, + "Tensor": { + "type": "object", + "properties": { + "shape": { + "type": "array", + "items": { + "type": "integer", + "format": "int32" + } + }, + "values": { + "type": "array", + "items": { + "type": "number", + "format": "double" + } + } + } + }, + "TensorShapeProtoDim": { + "type": "object", + "properties": { + "size": { + "type": "string", + "format": "int64", + "description": "Size of the tensor in that dimension.\nThis value must be >= -1, but values of -1 are reserved for \"unknown\"\nshapes (values of -1 mean \"unknown\" dimension). Certain wrappers\nthat work with TensorShapeProto may fail at runtime when deserializing\na TensorShapeProto containing a dim value of -1." + }, + "name": { + "type": "string", + "description": "Optional name of the tensor dimension." + } + }, + "description": "One dimension of the tensor." + }, + "TensorflowDataType": { + "type": "string", + "enum": [ + "DT_INVALID", + "DT_FLOAT", + "DT_DOUBLE", + "DT_INT32", + "DT_UINT8", + "DT_INT16", + "DT_INT8", + "DT_STRING", + "DT_COMPLEX64", + "DT_INT64", + "DT_BOOL", + "DT_QINT8", + "DT_QUINT8", + "DT_QINT32", + "DT_BFLOAT16", + "DT_QINT16", + "DT_QUINT16", + "DT_UINT16", + "DT_COMPLEX128", + "DT_HALF", + "DT_RESOURCE", + "DT_VARIANT", + "DT_UINT32", + "DT_UINT64", + "DT_FLOAT_REF", + "DT_DOUBLE_REF", + "DT_INT32_REF", + "DT_UINT8_REF", + "DT_INT16_REF", + "DT_INT8_REF", + "DT_STRING_REF", + "DT_COMPLEX64_REF", + "DT_INT64_REF", + "DT_BOOL_REF", + "DT_QINT8_REF", + "DT_QUINT8_REF", + "DT_QINT32_REF", + "DT_BFLOAT16_REF", + "DT_QINT16_REF", + "DT_QUINT16_REF", + "DT_UINT16_REF", + "DT_COMPLEX128_REF", + "DT_HALF_REF", + "DT_RESOURCE_REF", + "DT_VARIANT_REF", + "DT_UINT32_REF", + "DT_UINT64_REF" + ], + "default": "DT_INVALID", + "description": "- DT_INVALID: Not a legal value for DataType. Used to indicate a DataType field\nhas not been set.\n - DT_FLOAT: Data types that all computation devices are expected to be\ncapable to support.\n - DT_FLOAT_REF: Do not use! These are only for parameters. Every enum above\nshould have a corresponding value below (verified by types_test).", + "title": "LINT.IfChange" + }, + "TensorflowResourceHandleProto": { + "type": "object", + "properties": { + "device": { + "type": "string", + "description": "Unique name for the device containing the resource." + }, + "container": { + "type": "string", + "description": "Container in which this resource is placed." + }, + "name": { + "type": "string", + "description": "Unique name of this resource." + }, + "hash_code": { + "type": "string", + "format": "uint64", + "description": "Hash code for the type of the resource. Is only valid in the same device\nand in the same execution." + }, + "maybe_type_name": { + "type": "string", + "description": "For debug-only, the name of the type pointed to by this handle, if\navailable." + } + }, + "description": "Protocol buffer representing a handle to a tensorflow resource. Handles are\nnot valid across executions, but can be serialized back and forth from within\na single run." + }, + "TensorflowTensorProto": { + "type": "object", + "properties": { + "dtype": { + "$ref": "#/components/schemas/TensorflowDataType" + }, + "tensor_shape": { + "$ref": "#/components/schemas/TensorflowTensorShapeProto", + "description": "Shape of the tensor. TODO(touts): sort out the 0-rank issues." + }, + "version_number": { + "type": "integer", + "format": "int32", + "description": "Version number.\n\nIn version 0, if the \"repeated xxx\" representations contain only one\nelement, that element is repeated to fill the shape. This makes it easy\nto represent a constant Tensor with a single value." + }, + "tensor_content": { + "type": "string", + "format": "byte", + "description": "Serialized raw tensor content from either Tensor::AsProtoTensorContent or\nmemcpy in tensorflow::grpc::EncodeTensorToByteBuffer. This representation\ncan be used for all tensor types. The purpose of this representation is to\nreduce serialization overhead during RPC call by avoiding serialization of\nmany repeated small items." + }, + "half_val": { + "type": "array", + "items": { + "type": "integer", + "format": "int32" + }, + "description": "DT_HALF, DT_BFLOAT16. Note that since protobuf has no int16 type, we'll\nhave some pointless zero padding for each value here." + }, + "float_val": { + "type": "array", + "items": { + "type": "number", + "format": "float" + }, + "description": "DT_FLOAT." + }, + "double_val": { + "type": "array", + "items": { + "type": "number", + "format": "double" + }, + "description": "DT_DOUBLE." + }, + "int_val": { + "type": "array", + "items": { + "type": "integer", + "format": "int32" + }, + "description": "DT_INT32, DT_INT16, DT_INT8, DT_UINT8." + }, + "string_val": { + "type": "array", + "items": { + "type": "string", + "format": "byte" + }, + "title": "DT_STRING" + }, + "scomplex_val": { + "type": "array", + "items": { + "type": "number", + "format": "float" + }, + "description": "DT_COMPLEX64. scomplex_val(2*i) and scomplex_val(2*i+1) are real\nand imaginary parts of i-th single precision complex." + }, + "int64_val": { + "type": "array", + "items": { + "type": "string", + "format": "int64" + }, + "title": "DT_INT64" + }, + "bool_val": { + "type": "array", + "items": { + "type": "boolean", + "format": "boolean" + }, + "title": "DT_BOOL" + }, + "dcomplex_val": { + "type": "array", + "items": { + "type": "number", + "format": "double" + }, + "description": "DT_COMPLEX128. dcomplex_val(2*i) and dcomplex_val(2*i+1) are real\nand imaginary parts of i-th double precision complex." + }, + "resource_handle_val": { + "type": "array", + "items": { + "$ref": "#/components/schemas/TensorflowResourceHandleProto" + }, + "title": "DT_RESOURCE" + }, + "variant_val": { + "type": "array", + "items": { + "$ref": "#/components/schemas/TensorflowVariantTensorDataProto" + }, + "title": "DT_VARIANT" + }, + "uint32_val": { + "type": "array", + "items": { + "type": "integer", + "format": "int64" + }, + "title": "DT_UINT32" + }, + "uint64_val": { + "type": "array", + "items": { + "type": "string", + "format": "uint64" + }, + "title": "DT_UINT64" + } + }, + "description": "Protocol buffer representing a tensor." + }, + "TensorflowTensorShapeProto": { + "type": "object", + "properties": { + "dim": { + "type": "array", + "items": { + "$ref": "#/components/schemas/TensorShapeProtoDim" + }, + "description": "Dimensions of the tensor, such as {\"input\", 30}, {\"output\", 40}\nfor a 30 x 40 2D tensor. If an entry has size -1, this\ncorresponds to a dimension of unknown size. The names are\noptional.\n\nThe order of entries in \"dim\" matters: It indicates the layout of the\nvalues in the tensor in-memory representation.\n\nThe first entry in \"dim\" is the outermost dimension used to layout the\nvalues, the last entry is the innermost dimension. This matches the\nin-memory layout of RowMajor Eigen tensors.\n\nIf \"dim.size()\" > 0, \"unknown_rank\" must be false." + }, + "unknown_rank": { + "type": "boolean", + "format": "boolean", + "description": "If true, the number of dimensions in the shape is unknown.\n\nIf true, \"dim.size()\" must be 0." + } + }, + "description": "Dimensions of a tensor." + }, + "TensorflowVariantTensorDataProto": { + "type": "object", + "properties": { + "type_name": { + "type": "string", + "description": "Name of the type of objects being serialized." + }, + "metadata": { + "type": "string", + "format": "byte", + "description": "Portions of the object that are not Tensors." + }, + "tensors": { + "type": "array", + "items": { + "$ref": "#/components/schemas/TensorflowTensorProto" + }, + "description": "Tensors contained within objects being serialized." + } + }, + "description": "Protocol buffer representing the serialization format of DT_VARIANT tensors." + } + } + } +} \ No newline at end of file diff --git a/wrappers/python/outlier_detector_microservice.py b/python/seldon_core/outlier_detector_microservice.py similarity index 57% rename from wrappers/python/outlier_detector_microservice.py rename to python/seldon_core/outlier_detector_microservice.py index a2c90c14fe..aff2c83448 100644 --- a/wrappers/python/outlier_detector_microservice.py +++ b/python/seldon_core/outlier_detector_microservice.py @@ -1,59 +1,65 @@ -from proto import prediction_pb2, prediction_pb2_grpc -from microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ - array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ - SeldonMicroserviceException import grpc from concurrent import futures from flask import jsonify, Flask, send_from_directory from flask_cors import CORS import numpy as np +import logging + +from seldon_core.proto import prediction_pb2, prediction_pb2_grpc +from seldon_core.microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ + array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ + SeldonMicroserviceException + +logger = logging.getLogger(__name__) # --------------------------- # Interaction with user model # --------------------------- -def score(user_model,features,feature_names): + +def score(user_model, features, feature_names): # Returns a numpy array of floats that corresponds to the outlier scores for each point in the batch - return user_model.score(features,feature_names) - + return user_model.score(features, feature_names) + # ---------------------------- # REST # ---------------------------- -def get_rest_microservice(user_model,debug=False): - app = Flask(__name__,static_url_path='') +def get_rest_microservice(user_model, debug=False): + logger = logging.getLogger(__name__ + '.get_rest_microservice') + + app = Flask(__name__, static_url_path='') CORS(app) - + @app.errorhandler(SeldonMicroserviceException) def handle_invalid_usage(error): response = jsonify(error.to_dict()) - print("ERROR:") - print(error.to_dict()) + logger.error("%s", error.to_dict()) response.status_code = 400 return response - @app.route("/seldon.json",methods=["GET"]) + @app.route("/seldon.json", methods=["GET"]) def openAPI(): - return send_from_directory('', "seldon.json") + return send_from_directory("openapi", "seldon.json") - @app.route("/transform-input",methods=["GET","POST"]) + @app.route("/transform-input", methods=["GET", "POST"]) def TransformInput(): request = extract_message() sanity_check_request(request) - + datadef = request.get("data") features = rest_datadef_to_array(datadef) - outlier_scores = score(user_model,features,datadef.get("names")) + outlier_scores = score(user_model, features, datadef.get("names")) # TODO: check that predictions is 2 dimensional - request["meta"].setdefault("tags",{}) + request["meta"].setdefault("tags", {}) request["meta"]["tags"]["outlierScore"] = list(outlier_scores) return jsonify(request) - + return app @@ -62,28 +68,30 @@ def TransformInput(): # ---------------------------- class SeldonTransformerGRPC(object): - def __init__(self,user_model): + def __init__(self, user_model): self.user_model = user_model - def TransformInput(self,request,context): + def TransformInput(self, request, context): datadef = request.data features = grpc_datadef_to_array(datadef) - outlier_scores = score(self.user_model,features,datadef.names) + outlier_scores = score(self.user_model, features, datadef.names) request.meta.tags["outlierScore"] = list(outlier_scores) return request - -def get_grpc_server(user_model,debug=False,annotations={}): + + +def get_grpc_server(user_model, debug=False, annotations={}): seldon_model = SeldonTransformerGRPC(user_model) options = [] if ANNOTATION_GRPC_MAX_MSG_SIZE in annotations: max_msg = int(annotations[ANNOTATION_GRPC_MAX_MSG_SIZE]) - logger.info("Setting grpc max message to %d",max_msg) - options.append(('grpc.max_message_length', max_msg )) + logger.info("Setting grpc max message to %d", max_msg) + options.append(('grpc.max_message_length', max_msg)) - server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),options=options) + server = grpc.server(futures.ThreadPoolExecutor( + max_workers=10), options=options) prediction_pb2_grpc.add_ModelServicer_to_server(seldon_model, server) return server diff --git a/python/seldon_core/persistence.py b/python/seldon_core/persistence.py new file mode 100644 index 0000000000..d7151a8a03 --- /dev/null +++ b/python/seldon_core/persistence.py @@ -0,0 +1,64 @@ +import threading +import os +import time +import logging +try: + # python 2 + import cPickle as pickle +except ImportError: + # python 3 + import pickle +import redis + +logger = logging.getLogger(__name__) + +PRED_UNIT_ID = os.environ.get("PREDICTIVE_UNIT_ID", "0") +PREDICTOR_ID = os.environ.get("PREDICTOR_ID", "0") +DEPLOYMENT_ID = os.environ.get("SELDON_DEPLOYMENT_ID", "0") +REDIS_KEY = "persistence_{}_{}_{}".format( + DEPLOYMENT_ID, PREDICTOR_ID, PRED_UNIT_ID) + +REDIS_HOST = os.environ.get('REDIS_SERVICE_HOST', 'localhost') +REDIS_PORT = os.environ.get("REDIS_SERVICE_PORT", 6379) +DEFAULT_PUSH_FREQUENCY = 60 + + +def restore(user_class, parameters, debug=False): + logger.info("Restoring saved model from redis") + + redis_client = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT) + saved_state_binary = redis_client.get(REDIS_KEY) + if saved_state_binary is None: + logger.info("Saved state is empty, restoration aborted") + return user_class(**parameters) + else: + return pickle.loads(saved_state_binary) + + +def persist(user_object, push_frequency=None, debug=False): + + if push_frequency is None: + push_frequency = DEFAULT_PUSH_FREQUENCY + logger.info("Creating persistence thread, with frequency %s", push_frequency) + persistence_thread = PersistenceThread(user_object, push_frequency) + persistence_thread.start() + + +class PersistenceThread(threading.Thread): + + def __init__(self, user_object, push_frequency): + self.user_object = user_object + self.push_frequency = push_frequency + self._stopped = False + self.redis_client = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT) + super(PersistenceThread, self).__init__() + + def stop(self): + logger.info("Stopping Persistence Thread") + self._stopped = True + + def run(self): + while not self._stopped: + time.sleep(self.push_frequency) + binary_data = pickle.dumps(self.user_object) + self.redis_client.set(REDIS_KEY, binary_data) diff --git a/wrappers/python/proto/__init__.py b/python/seldon_core/proto/__init__.py similarity index 100% rename from wrappers/python/proto/__init__.py rename to python/seldon_core/proto/__init__.py diff --git a/python/seldon_core/proto/prediction.proto b/python/seldon_core/proto/prediction.proto new file mode 100644 index 0000000000..2d2f3afae5 --- /dev/null +++ b/python/seldon_core/proto/prediction.proto @@ -0,0 +1,128 @@ +syntax = "proto3"; + +import "google/protobuf/struct.proto"; +import "tensorflow/core/framework/tensor.proto"; + +package seldon.protos; + +option java_package = "io.seldon.protos"; +option java_outer_classname = "PredictionProtos"; + +// [START Messages] + +message SeldonMessage { + + Status status = 1; + Meta meta = 2; + oneof data_oneof { + DefaultData data = 3; + bytes binData = 4; + string strData = 5; + } +} + +message DefaultData { + repeated string names = 1; + oneof data_oneof { + Tensor tensor = 2; + google.protobuf.ListValue ndarray = 3; + tensorflow.TensorProto tftensor = 4; + } +} + +message Tensor { + repeated int32 shape = 1 [packed=true]; + repeated double values = 2 [packed=true]; +} + +message Meta { + string puid = 1; + map tags = 2; + map routing = 3; + map requestPath = 4; + repeated Metric metrics = 5; +} + +message Metric { + enum MetricType { + COUNTER = 0; + GAUGE = 1; + TIMER = 2; + } + string key = 1; + MetricType type = 2; + float value = 3; + map tags = 4; +} + +message SeldonMessageList { + repeated SeldonMessage seldonMessages = 1; +} + +message Status { + + enum StatusFlag { + SUCCESS = 0; + FAILURE = 1; + } + + int32 code = 1; + string info = 2; + string reason = 3; + StatusFlag status = 4; +} + +message Feedback { + SeldonMessage request = 1; + SeldonMessage response = 2; + float reward = 3; + SeldonMessage truth = 4; +} + +message RequestResponse { + SeldonMessage request = 1; + SeldonMessage response = 2; +} + +// [END Messages] + + +// [START Services] + +service Generic { + rpc TransformInput(SeldonMessage) returns (SeldonMessage) {}; + rpc TransformOutput(SeldonMessage) returns (SeldonMessage) {}; + rpc Route(SeldonMessage) returns (SeldonMessage) {}; + rpc Aggregate(SeldonMessageList) returns (SeldonMessage) {}; + rpc SendFeedback(Feedback) returns (SeldonMessage) {}; +} + +service Model { + rpc Predict(SeldonMessage) returns (SeldonMessage) {}; + rpc SendFeedback(Feedback) returns (SeldonMessage) {}; + } + +service Router { + rpc Route(SeldonMessage) returns (SeldonMessage) {}; + rpc SendFeedback(Feedback) returns (SeldonMessage) {}; + } + +service Transformer { + rpc TransformInput(SeldonMessage) returns (SeldonMessage) {}; +} + +service OutputTransformer { + rpc TransformOutput(SeldonMessage) returns (SeldonMessage) {}; +} + +service Combiner { + rpc Aggregate(SeldonMessageList) returns (SeldonMessage) {}; +} + + +service Seldon { + rpc Predict(SeldonMessage) returns (SeldonMessage) {}; + rpc SendFeedback(Feedback) returns (SeldonMessage) {}; + } + +// [END Services] \ No newline at end of file diff --git a/python/seldon_core/proto/prediction_pb2.py b/python/seldon_core/proto/prediction_pb2.py new file mode 100644 index 0000000000..c8af25de66 --- /dev/null +++ b/python/seldon_core/proto/prediction_pb2.py @@ -0,0 +1,1056 @@ +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: proto/prediction.proto + +import sys +_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from google.protobuf import reflection as _reflection +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + +from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2 +from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2 + + +DESCRIPTOR = _descriptor.FileDescriptor( + name='proto/prediction.proto', + package='seldon.protos', + syntax='proto3', + serialized_options=_b('\n\020io.seldon.protosB\020PredictionProtos'), + serialized_pb=_b('\n\x16proto/prediction.proto\x12\rseldon.protos\x1a\x1cgoogle/protobuf/struct.proto\x1a&tensorflow/core/framework/tensor.proto\"\xb9\x01\n\rSeldonMessage\x12%\n\x06status\x18\x01 \x01(\x0b\x32\x15.seldon.protos.Status\x12!\n\x04meta\x18\x02 \x01(\x0b\x32\x13.seldon.protos.Meta\x12*\n\x04\x64\x61ta\x18\x03 \x01(\x0b\x32\x1a.seldon.protos.DefaultDataH\x00\x12\x11\n\x07\x62inData\x18\x04 \x01(\x0cH\x00\x12\x11\n\x07strData\x18\x05 \x01(\tH\x00\x42\x0c\n\ndata_oneof\"\xaf\x01\n\x0b\x44\x65\x66\x61ultData\x12\r\n\x05names\x18\x01 \x03(\t\x12\'\n\x06tensor\x18\x02 \x01(\x0b\x32\x15.seldon.protos.TensorH\x00\x12-\n\x07ndarray\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.ListValueH\x00\x12+\n\x08tftensor\x18\x04 \x01(\x0b\x32\x17.tensorflow.TensorProtoH\x00\x42\x0c\n\ndata_oneof\"/\n\x06Tensor\x12\x11\n\x05shape\x18\x01 \x03(\x05\x42\x02\x10\x01\x12\x12\n\x06values\x18\x02 \x03(\x01\x42\x02\x10\x01\"\x80\x03\n\x04Meta\x12\x0c\n\x04puid\x18\x01 \x01(\t\x12+\n\x04tags\x18\x02 \x03(\x0b\x32\x1d.seldon.protos.Meta.TagsEntry\x12\x31\n\x07routing\x18\x03 \x03(\x0b\x32 .seldon.protos.Meta.RoutingEntry\x12\x39\n\x0brequestPath\x18\x04 \x03(\x0b\x32$.seldon.protos.Meta.RequestPathEntry\x12&\n\x07metrics\x18\x05 \x03(\x0b\x32\x15.seldon.protos.Metric\x1a\x43\n\tTagsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12%\n\x05value\x18\x02 \x01(\x0b\x32\x16.google.protobuf.Value:\x02\x38\x01\x1a.\n\x0cRoutingEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x05:\x02\x38\x01\x1a\x32\n\x10RequestPathEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xe1\x01\n\x06Metric\x12\x0b\n\x03key\x18\x01 \x01(\t\x12.\n\x04type\x18\x02 \x01(\x0e\x32 .seldon.protos.Metric.MetricType\x12\r\n\x05value\x18\x03 \x01(\x02\x12-\n\x04tags\x18\x04 \x03(\x0b\x32\x1f.seldon.protos.Metric.TagsEntry\x1a+\n\tTagsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"/\n\nMetricType\x12\x0b\n\x07\x43OUNTER\x10\x00\x12\t\n\x05GAUGE\x10\x01\x12\t\n\x05TIMER\x10\x02\"I\n\x11SeldonMessageList\x12\x34\n\x0eseldonMessages\x18\x01 \x03(\x0b\x32\x1c.seldon.protos.SeldonMessage\"\x8e\x01\n\x06Status\x12\x0c\n\x04\x63ode\x18\x01 \x01(\x05\x12\x0c\n\x04info\x18\x02 \x01(\t\x12\x0e\n\x06reason\x18\x03 \x01(\t\x12\x30\n\x06status\x18\x04 \x01(\x0e\x32 .seldon.protos.Status.StatusFlag\"&\n\nStatusFlag\x12\x0b\n\x07SUCCESS\x10\x00\x12\x0b\n\x07\x46\x41ILURE\x10\x01\"\xa6\x01\n\x08\x46\x65\x65\x64\x62\x61\x63k\x12-\n\x07request\x18\x01 \x01(\x0b\x32\x1c.seldon.protos.SeldonMessage\x12.\n\x08response\x18\x02 \x01(\x0b\x32\x1c.seldon.protos.SeldonMessage\x12\x0e\n\x06reward\x18\x03 \x01(\x02\x12+\n\x05truth\x18\x04 \x01(\x0b\x32\x1c.seldon.protos.SeldonMessage\"p\n\x0fRequestResponse\x12-\n\x07request\x18\x01 \x01(\x0b\x32\x1c.seldon.protos.SeldonMessage\x12.\n\x08response\x18\x02 \x01(\x0b\x32\x1c.seldon.protos.SeldonMessage2\x89\x03\n\x07Generic\x12N\n\x0eTransformInput\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12O\n\x0fTransformOutput\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12\x45\n\x05Route\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12M\n\tAggregate\x12 .seldon.protos.SeldonMessageList\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12G\n\x0cSendFeedback\x12\x17.seldon.protos.Feedback\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32\x99\x01\n\x05Model\x12G\n\x07Predict\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12G\n\x0cSendFeedback\x12\x17.seldon.protos.Feedback\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32\x98\x01\n\x06Router\x12\x45\n\x05Route\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12G\n\x0cSendFeedback\x12\x17.seldon.protos.Feedback\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32]\n\x0bTransformer\x12N\n\x0eTransformInput\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32\x64\n\x11OutputTransformer\x12O\n\x0fTransformOutput\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32Y\n\x08\x43ombiner\x12M\n\tAggregate\x12 .seldon.protos.SeldonMessageList\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x32\x9a\x01\n\x06Seldon\x12G\n\x07Predict\x12\x1c.seldon.protos.SeldonMessage\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x12G\n\x0cSendFeedback\x12\x17.seldon.protos.Feedback\x1a\x1c.seldon.protos.SeldonMessage\"\x00\x42$\n\x10io.seldon.protosB\x10PredictionProtosb\x06proto3') + , + dependencies=[google_dot_protobuf_dot_struct__pb2.DESCRIPTOR,tensorflow_dot_core_dot_framework_dot_tensor__pb2.DESCRIPTOR,]) + + + +_METRIC_METRICTYPE = _descriptor.EnumDescriptor( + name='MetricType', + full_name='seldon.protos.Metric.MetricType', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='COUNTER', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='GAUGE', index=1, number=1, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='TIMER', index=2, number=2, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=1092, + serialized_end=1139, +) +_sym_db.RegisterEnumDescriptor(_METRIC_METRICTYPE) + +_STATUS_STATUSFLAG = _descriptor.EnumDescriptor( + name='StatusFlag', + full_name='seldon.protos.Status.StatusFlag', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='SUCCESS', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='FAILURE', index=1, number=1, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=1321, + serialized_end=1359, +) +_sym_db.RegisterEnumDescriptor(_STATUS_STATUSFLAG) + + +_SELDONMESSAGE = _descriptor.Descriptor( + name='SeldonMessage', + full_name='seldon.protos.SeldonMessage', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='status', full_name='seldon.protos.SeldonMessage.status', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='meta', full_name='seldon.protos.SeldonMessage.meta', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='data', full_name='seldon.protos.SeldonMessage.data', index=2, + number=3, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='binData', full_name='seldon.protos.SeldonMessage.binData', index=3, + number=4, type=12, cpp_type=9, label=1, + has_default_value=False, default_value=_b(""), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='strData', full_name='seldon.protos.SeldonMessage.strData', index=4, + number=5, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='data_oneof', full_name='seldon.protos.SeldonMessage.data_oneof', + index=0, containing_type=None, fields=[]), + ], + serialized_start=112, + serialized_end=297, +) + + +_DEFAULTDATA = _descriptor.Descriptor( + name='DefaultData', + full_name='seldon.protos.DefaultData', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='names', full_name='seldon.protos.DefaultData.names', index=0, + number=1, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='tensor', full_name='seldon.protos.DefaultData.tensor', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='ndarray', full_name='seldon.protos.DefaultData.ndarray', index=2, + number=3, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='tftensor', full_name='seldon.protos.DefaultData.tftensor', index=3, + number=4, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='data_oneof', full_name='seldon.protos.DefaultData.data_oneof', + index=0, containing_type=None, fields=[]), + ], + serialized_start=300, + serialized_end=475, +) + + +_TENSOR = _descriptor.Descriptor( + name='Tensor', + full_name='seldon.protos.Tensor', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='shape', full_name='seldon.protos.Tensor.shape', index=0, + number=1, type=5, cpp_type=1, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=_b('\020\001'), file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='values', full_name='seldon.protos.Tensor.values', index=1, + number=2, type=1, cpp_type=5, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=_b('\020\001'), file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=477, + serialized_end=524, +) + + +_META_TAGSENTRY = _descriptor.Descriptor( + name='TagsEntry', + full_name='seldon.protos.Meta.TagsEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='seldon.protos.Meta.TagsEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='seldon.protos.Meta.TagsEntry.value', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=_b('8\001'), + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=744, + serialized_end=811, +) + +_META_ROUTINGENTRY = _descriptor.Descriptor( + name='RoutingEntry', + full_name='seldon.protos.Meta.RoutingEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='seldon.protos.Meta.RoutingEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='seldon.protos.Meta.RoutingEntry.value', index=1, + number=2, type=5, cpp_type=1, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=_b('8\001'), + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=813, + serialized_end=859, +) + +_META_REQUESTPATHENTRY = _descriptor.Descriptor( + name='RequestPathEntry', + full_name='seldon.protos.Meta.RequestPathEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='seldon.protos.Meta.RequestPathEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='seldon.protos.Meta.RequestPathEntry.value', index=1, + number=2, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=_b('8\001'), + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=861, + serialized_end=911, +) + +_META = _descriptor.Descriptor( + name='Meta', + full_name='seldon.protos.Meta', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='puid', full_name='seldon.protos.Meta.puid', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='tags', full_name='seldon.protos.Meta.tags', index=1, + number=2, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='routing', full_name='seldon.protos.Meta.routing', index=2, + number=3, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='requestPath', full_name='seldon.protos.Meta.requestPath', index=3, + number=4, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='metrics', full_name='seldon.protos.Meta.metrics', index=4, + number=5, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_META_TAGSENTRY, _META_ROUTINGENTRY, _META_REQUESTPATHENTRY, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=527, + serialized_end=911, +) + + +_METRIC_TAGSENTRY = _descriptor.Descriptor( + name='TagsEntry', + full_name='seldon.protos.Metric.TagsEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='seldon.protos.Metric.TagsEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='seldon.protos.Metric.TagsEntry.value', index=1, + number=2, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=_b('8\001'), + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=1047, + serialized_end=1090, +) + +_METRIC = _descriptor.Descriptor( + name='Metric', + full_name='seldon.protos.Metric', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='seldon.protos.Metric.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='type', full_name='seldon.protos.Metric.type', index=1, + number=2, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='seldon.protos.Metric.value', index=2, + number=3, type=2, cpp_type=6, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='tags', full_name='seldon.protos.Metric.tags', index=3, + number=4, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_METRIC_TAGSENTRY, ], + enum_types=[ + _METRIC_METRICTYPE, + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=914, + serialized_end=1139, +) + + +_SELDONMESSAGELIST = _descriptor.Descriptor( + name='SeldonMessageList', + full_name='seldon.protos.SeldonMessageList', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='seldonMessages', full_name='seldon.protos.SeldonMessageList.seldonMessages', index=0, + number=1, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=1141, + serialized_end=1214, +) + + +_STATUS = _descriptor.Descriptor( + name='Status', + full_name='seldon.protos.Status', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='code', full_name='seldon.protos.Status.code', index=0, + number=1, type=5, cpp_type=1, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='info', full_name='seldon.protos.Status.info', index=1, + number=2, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='reason', full_name='seldon.protos.Status.reason', index=2, + number=3, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=_b("").decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='status', full_name='seldon.protos.Status.status', index=3, + number=4, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + _STATUS_STATUSFLAG, + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=1217, + serialized_end=1359, +) + + +_FEEDBACK = _descriptor.Descriptor( + name='Feedback', + full_name='seldon.protos.Feedback', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='request', full_name='seldon.protos.Feedback.request', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='response', full_name='seldon.protos.Feedback.response', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='reward', full_name='seldon.protos.Feedback.reward', index=2, + number=3, type=2, cpp_type=6, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='truth', full_name='seldon.protos.Feedback.truth', index=3, + number=4, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=1362, + serialized_end=1528, +) + + +_REQUESTRESPONSE = _descriptor.Descriptor( + name='RequestResponse', + full_name='seldon.protos.RequestResponse', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='request', full_name='seldon.protos.RequestResponse.request', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='response', full_name='seldon.protos.RequestResponse.response', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto3', + extension_ranges=[], + oneofs=[ + ], + serialized_start=1530, + serialized_end=1642, +) + +_SELDONMESSAGE.fields_by_name['status'].message_type = _STATUS +_SELDONMESSAGE.fields_by_name['meta'].message_type = _META +_SELDONMESSAGE.fields_by_name['data'].message_type = _DEFAULTDATA +_SELDONMESSAGE.oneofs_by_name['data_oneof'].fields.append( + _SELDONMESSAGE.fields_by_name['data']) +_SELDONMESSAGE.fields_by_name['data'].containing_oneof = _SELDONMESSAGE.oneofs_by_name['data_oneof'] +_SELDONMESSAGE.oneofs_by_name['data_oneof'].fields.append( + _SELDONMESSAGE.fields_by_name['binData']) +_SELDONMESSAGE.fields_by_name['binData'].containing_oneof = _SELDONMESSAGE.oneofs_by_name['data_oneof'] +_SELDONMESSAGE.oneofs_by_name['data_oneof'].fields.append( + _SELDONMESSAGE.fields_by_name['strData']) +_SELDONMESSAGE.fields_by_name['strData'].containing_oneof = _SELDONMESSAGE.oneofs_by_name['data_oneof'] +_DEFAULTDATA.fields_by_name['tensor'].message_type = _TENSOR +_DEFAULTDATA.fields_by_name['ndarray'].message_type = google_dot_protobuf_dot_struct__pb2._LISTVALUE +_DEFAULTDATA.fields_by_name['tftensor'].message_type = tensorflow_dot_core_dot_framework_dot_tensor__pb2._TENSORPROTO +_DEFAULTDATA.oneofs_by_name['data_oneof'].fields.append( + _DEFAULTDATA.fields_by_name['tensor']) +_DEFAULTDATA.fields_by_name['tensor'].containing_oneof = _DEFAULTDATA.oneofs_by_name['data_oneof'] +_DEFAULTDATA.oneofs_by_name['data_oneof'].fields.append( + _DEFAULTDATA.fields_by_name['ndarray']) +_DEFAULTDATA.fields_by_name['ndarray'].containing_oneof = _DEFAULTDATA.oneofs_by_name['data_oneof'] +_DEFAULTDATA.oneofs_by_name['data_oneof'].fields.append( + _DEFAULTDATA.fields_by_name['tftensor']) +_DEFAULTDATA.fields_by_name['tftensor'].containing_oneof = _DEFAULTDATA.oneofs_by_name['data_oneof'] +_META_TAGSENTRY.fields_by_name['value'].message_type = google_dot_protobuf_dot_struct__pb2._VALUE +_META_TAGSENTRY.containing_type = _META +_META_ROUTINGENTRY.containing_type = _META +_META_REQUESTPATHENTRY.containing_type = _META +_META.fields_by_name['tags'].message_type = _META_TAGSENTRY +_META.fields_by_name['routing'].message_type = _META_ROUTINGENTRY +_META.fields_by_name['requestPath'].message_type = _META_REQUESTPATHENTRY +_META.fields_by_name['metrics'].message_type = _METRIC +_METRIC_TAGSENTRY.containing_type = _METRIC +_METRIC.fields_by_name['type'].enum_type = _METRIC_METRICTYPE +_METRIC.fields_by_name['tags'].message_type = _METRIC_TAGSENTRY +_METRIC_METRICTYPE.containing_type = _METRIC +_SELDONMESSAGELIST.fields_by_name['seldonMessages'].message_type = _SELDONMESSAGE +_STATUS.fields_by_name['status'].enum_type = _STATUS_STATUSFLAG +_STATUS_STATUSFLAG.containing_type = _STATUS +_FEEDBACK.fields_by_name['request'].message_type = _SELDONMESSAGE +_FEEDBACK.fields_by_name['response'].message_type = _SELDONMESSAGE +_FEEDBACK.fields_by_name['truth'].message_type = _SELDONMESSAGE +_REQUESTRESPONSE.fields_by_name['request'].message_type = _SELDONMESSAGE +_REQUESTRESPONSE.fields_by_name['response'].message_type = _SELDONMESSAGE +DESCRIPTOR.message_types_by_name['SeldonMessage'] = _SELDONMESSAGE +DESCRIPTOR.message_types_by_name['DefaultData'] = _DEFAULTDATA +DESCRIPTOR.message_types_by_name['Tensor'] = _TENSOR +DESCRIPTOR.message_types_by_name['Meta'] = _META +DESCRIPTOR.message_types_by_name['Metric'] = _METRIC +DESCRIPTOR.message_types_by_name['SeldonMessageList'] = _SELDONMESSAGELIST +DESCRIPTOR.message_types_by_name['Status'] = _STATUS +DESCRIPTOR.message_types_by_name['Feedback'] = _FEEDBACK +DESCRIPTOR.message_types_by_name['RequestResponse'] = _REQUESTRESPONSE +_sym_db.RegisterFileDescriptor(DESCRIPTOR) + +SeldonMessage = _reflection.GeneratedProtocolMessageType('SeldonMessage', (_message.Message,), dict( + DESCRIPTOR = _SELDONMESSAGE, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.SeldonMessage) + )) +_sym_db.RegisterMessage(SeldonMessage) + +DefaultData = _reflection.GeneratedProtocolMessageType('DefaultData', (_message.Message,), dict( + DESCRIPTOR = _DEFAULTDATA, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.DefaultData) + )) +_sym_db.RegisterMessage(DefaultData) + +Tensor = _reflection.GeneratedProtocolMessageType('Tensor', (_message.Message,), dict( + DESCRIPTOR = _TENSOR, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Tensor) + )) +_sym_db.RegisterMessage(Tensor) + +Meta = _reflection.GeneratedProtocolMessageType('Meta', (_message.Message,), dict( + + TagsEntry = _reflection.GeneratedProtocolMessageType('TagsEntry', (_message.Message,), dict( + DESCRIPTOR = _META_TAGSENTRY, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Meta.TagsEntry) + )) + , + + RoutingEntry = _reflection.GeneratedProtocolMessageType('RoutingEntry', (_message.Message,), dict( + DESCRIPTOR = _META_ROUTINGENTRY, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Meta.RoutingEntry) + )) + , + + RequestPathEntry = _reflection.GeneratedProtocolMessageType('RequestPathEntry', (_message.Message,), dict( + DESCRIPTOR = _META_REQUESTPATHENTRY, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Meta.RequestPathEntry) + )) + , + DESCRIPTOR = _META, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Meta) + )) +_sym_db.RegisterMessage(Meta) +_sym_db.RegisterMessage(Meta.TagsEntry) +_sym_db.RegisterMessage(Meta.RoutingEntry) +_sym_db.RegisterMessage(Meta.RequestPathEntry) + +Metric = _reflection.GeneratedProtocolMessageType('Metric', (_message.Message,), dict( + + TagsEntry = _reflection.GeneratedProtocolMessageType('TagsEntry', (_message.Message,), dict( + DESCRIPTOR = _METRIC_TAGSENTRY, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Metric.TagsEntry) + )) + , + DESCRIPTOR = _METRIC, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Metric) + )) +_sym_db.RegisterMessage(Metric) +_sym_db.RegisterMessage(Metric.TagsEntry) + +SeldonMessageList = _reflection.GeneratedProtocolMessageType('SeldonMessageList', (_message.Message,), dict( + DESCRIPTOR = _SELDONMESSAGELIST, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.SeldonMessageList) + )) +_sym_db.RegisterMessage(SeldonMessageList) + +Status = _reflection.GeneratedProtocolMessageType('Status', (_message.Message,), dict( + DESCRIPTOR = _STATUS, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Status) + )) +_sym_db.RegisterMessage(Status) + +Feedback = _reflection.GeneratedProtocolMessageType('Feedback', (_message.Message,), dict( + DESCRIPTOR = _FEEDBACK, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.Feedback) + )) +_sym_db.RegisterMessage(Feedback) + +RequestResponse = _reflection.GeneratedProtocolMessageType('RequestResponse', (_message.Message,), dict( + DESCRIPTOR = _REQUESTRESPONSE, + __module__ = 'proto.prediction_pb2' + # @@protoc_insertion_point(class_scope:seldon.protos.RequestResponse) + )) +_sym_db.RegisterMessage(RequestResponse) + + +DESCRIPTOR._options = None +_TENSOR.fields_by_name['shape']._options = None +_TENSOR.fields_by_name['values']._options = None +_META_TAGSENTRY._options = None +_META_ROUTINGENTRY._options = None +_META_REQUESTPATHENTRY._options = None +_METRIC_TAGSENTRY._options = None + +_GENERIC = _descriptor.ServiceDescriptor( + name='Generic', + full_name='seldon.protos.Generic', + file=DESCRIPTOR, + index=0, + serialized_options=None, + serialized_start=1645, + serialized_end=2038, + methods=[ + _descriptor.MethodDescriptor( + name='TransformInput', + full_name='seldon.protos.Generic.TransformInput', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='TransformOutput', + full_name='seldon.protos.Generic.TransformOutput', + index=1, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='Route', + full_name='seldon.protos.Generic.Route', + index=2, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='Aggregate', + full_name='seldon.protos.Generic.Aggregate', + index=3, + containing_service=None, + input_type=_SELDONMESSAGELIST, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='SendFeedback', + full_name='seldon.protos.Generic.SendFeedback', + index=4, + containing_service=None, + input_type=_FEEDBACK, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_GENERIC) + +DESCRIPTOR.services_by_name['Generic'] = _GENERIC + + +_MODEL = _descriptor.ServiceDescriptor( + name='Model', + full_name='seldon.protos.Model', + file=DESCRIPTOR, + index=1, + serialized_options=None, + serialized_start=2041, + serialized_end=2194, + methods=[ + _descriptor.MethodDescriptor( + name='Predict', + full_name='seldon.protos.Model.Predict', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='SendFeedback', + full_name='seldon.protos.Model.SendFeedback', + index=1, + containing_service=None, + input_type=_FEEDBACK, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_MODEL) + +DESCRIPTOR.services_by_name['Model'] = _MODEL + + +_ROUTER = _descriptor.ServiceDescriptor( + name='Router', + full_name='seldon.protos.Router', + file=DESCRIPTOR, + index=2, + serialized_options=None, + serialized_start=2197, + serialized_end=2349, + methods=[ + _descriptor.MethodDescriptor( + name='Route', + full_name='seldon.protos.Router.Route', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='SendFeedback', + full_name='seldon.protos.Router.SendFeedback', + index=1, + containing_service=None, + input_type=_FEEDBACK, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_ROUTER) + +DESCRIPTOR.services_by_name['Router'] = _ROUTER + + +_TRANSFORMER = _descriptor.ServiceDescriptor( + name='Transformer', + full_name='seldon.protos.Transformer', + file=DESCRIPTOR, + index=3, + serialized_options=None, + serialized_start=2351, + serialized_end=2444, + methods=[ + _descriptor.MethodDescriptor( + name='TransformInput', + full_name='seldon.protos.Transformer.TransformInput', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_TRANSFORMER) + +DESCRIPTOR.services_by_name['Transformer'] = _TRANSFORMER + + +_OUTPUTTRANSFORMER = _descriptor.ServiceDescriptor( + name='OutputTransformer', + full_name='seldon.protos.OutputTransformer', + file=DESCRIPTOR, + index=4, + serialized_options=None, + serialized_start=2446, + serialized_end=2546, + methods=[ + _descriptor.MethodDescriptor( + name='TransformOutput', + full_name='seldon.protos.OutputTransformer.TransformOutput', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_OUTPUTTRANSFORMER) + +DESCRIPTOR.services_by_name['OutputTransformer'] = _OUTPUTTRANSFORMER + + +_COMBINER = _descriptor.ServiceDescriptor( + name='Combiner', + full_name='seldon.protos.Combiner', + file=DESCRIPTOR, + index=5, + serialized_options=None, + serialized_start=2548, + serialized_end=2637, + methods=[ + _descriptor.MethodDescriptor( + name='Aggregate', + full_name='seldon.protos.Combiner.Aggregate', + index=0, + containing_service=None, + input_type=_SELDONMESSAGELIST, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_COMBINER) + +DESCRIPTOR.services_by_name['Combiner'] = _COMBINER + + +_SELDON = _descriptor.ServiceDescriptor( + name='Seldon', + full_name='seldon.protos.Seldon', + file=DESCRIPTOR, + index=6, + serialized_options=None, + serialized_start=2640, + serialized_end=2794, + methods=[ + _descriptor.MethodDescriptor( + name='Predict', + full_name='seldon.protos.Seldon.Predict', + index=0, + containing_service=None, + input_type=_SELDONMESSAGE, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), + _descriptor.MethodDescriptor( + name='SendFeedback', + full_name='seldon.protos.Seldon.SendFeedback', + index=1, + containing_service=None, + input_type=_FEEDBACK, + output_type=_SELDONMESSAGE, + serialized_options=None, + ), +]) +_sym_db.RegisterServiceDescriptor(_SELDON) + +DESCRIPTOR.services_by_name['Seldon'] = _SELDON + +# @@protoc_insertion_point(module_scope) diff --git a/python/seldon_core/proto/prediction_pb2_grpc.py b/python/seldon_core/proto/prediction_pb2_grpc.py new file mode 100644 index 0000000000..a8072d48a2 --- /dev/null +++ b/python/seldon_core/proto/prediction_pb2_grpc.py @@ -0,0 +1,423 @@ +# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! +import grpc + +from seldon_core.proto import prediction_pb2 as proto_dot_prediction__pb2 + + +class GenericStub(object): + """[END Messages] + + [START Services] + + """ + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.TransformInput = channel.unary_unary( + '/seldon.protos.Generic/TransformInput', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.TransformOutput = channel.unary_unary( + '/seldon.protos.Generic/TransformOutput', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.Route = channel.unary_unary( + '/seldon.protos.Generic/Route', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.Aggregate = channel.unary_unary( + '/seldon.protos.Generic/Aggregate', + request_serializer=proto_dot_prediction__pb2.SeldonMessageList.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.SendFeedback = channel.unary_unary( + '/seldon.protos.Generic/SendFeedback', + request_serializer=proto_dot_prediction__pb2.Feedback.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class GenericServicer(object): + """[END Messages] + + [START Services] + + """ + + def TransformInput(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def TransformOutput(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def Route(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def Aggregate(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def SendFeedback(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_GenericServicer_to_server(servicer, server): + rpc_method_handlers = { + 'TransformInput': grpc.unary_unary_rpc_method_handler( + servicer.TransformInput, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'TransformOutput': grpc.unary_unary_rpc_method_handler( + servicer.TransformOutput, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'Route': grpc.unary_unary_rpc_method_handler( + servicer.Route, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'Aggregate': grpc.unary_unary_rpc_method_handler( + servicer.Aggregate, + request_deserializer=proto_dot_prediction__pb2.SeldonMessageList.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'SendFeedback': grpc.unary_unary_rpc_method_handler( + servicer.SendFeedback, + request_deserializer=proto_dot_prediction__pb2.Feedback.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Generic', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class ModelStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Predict = channel.unary_unary( + '/seldon.protos.Model/Predict', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.SendFeedback = channel.unary_unary( + '/seldon.protos.Model/SendFeedback', + request_serializer=proto_dot_prediction__pb2.Feedback.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class ModelServicer(object): + # missing associated documentation comment in .proto file + pass + + def Predict(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def SendFeedback(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_ModelServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Predict': grpc.unary_unary_rpc_method_handler( + servicer.Predict, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'SendFeedback': grpc.unary_unary_rpc_method_handler( + servicer.SendFeedback, + request_deserializer=proto_dot_prediction__pb2.Feedback.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Model', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class RouterStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Route = channel.unary_unary( + '/seldon.protos.Router/Route', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.SendFeedback = channel.unary_unary( + '/seldon.protos.Router/SendFeedback', + request_serializer=proto_dot_prediction__pb2.Feedback.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class RouterServicer(object): + # missing associated documentation comment in .proto file + pass + + def Route(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def SendFeedback(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_RouterServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Route': grpc.unary_unary_rpc_method_handler( + servicer.Route, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'SendFeedback': grpc.unary_unary_rpc_method_handler( + servicer.SendFeedback, + request_deserializer=proto_dot_prediction__pb2.Feedback.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Router', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class TransformerStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.TransformInput = channel.unary_unary( + '/seldon.protos.Transformer/TransformInput', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class TransformerServicer(object): + # missing associated documentation comment in .proto file + pass + + def TransformInput(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_TransformerServicer_to_server(servicer, server): + rpc_method_handlers = { + 'TransformInput': grpc.unary_unary_rpc_method_handler( + servicer.TransformInput, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Transformer', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class OutputTransformerStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.TransformOutput = channel.unary_unary( + '/seldon.protos.OutputTransformer/TransformOutput', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class OutputTransformerServicer(object): + # missing associated documentation comment in .proto file + pass + + def TransformOutput(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_OutputTransformerServicer_to_server(servicer, server): + rpc_method_handlers = { + 'TransformOutput': grpc.unary_unary_rpc_method_handler( + servicer.TransformOutput, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.OutputTransformer', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class CombinerStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Aggregate = channel.unary_unary( + '/seldon.protos.Combiner/Aggregate', + request_serializer=proto_dot_prediction__pb2.SeldonMessageList.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class CombinerServicer(object): + # missing associated documentation comment in .proto file + pass + + def Aggregate(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_CombinerServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Aggregate': grpc.unary_unary_rpc_method_handler( + servicer.Aggregate, + request_deserializer=proto_dot_prediction__pb2.SeldonMessageList.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Combiner', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + +class SeldonStub(object): + # missing associated documentation comment in .proto file + pass + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Predict = channel.unary_unary( + '/seldon.protos.Seldon/Predict', + request_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + self.SendFeedback = channel.unary_unary( + '/seldon.protos.Seldon/SendFeedback', + request_serializer=proto_dot_prediction__pb2.Feedback.SerializeToString, + response_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + ) + + +class SeldonServicer(object): + # missing associated documentation comment in .proto file + pass + + def Predict(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def SendFeedback(self, request, context): + # missing associated documentation comment in .proto file + pass + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_SeldonServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Predict': grpc.unary_unary_rpc_method_handler( + servicer.Predict, + request_deserializer=proto_dot_prediction__pb2.SeldonMessage.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + 'SendFeedback': grpc.unary_unary_rpc_method_handler( + servicer.SendFeedback, + request_deserializer=proto_dot_prediction__pb2.Feedback.FromString, + response_serializer=proto_dot_prediction__pb2.SeldonMessage.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'seldon.protos.Seldon', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) diff --git a/wrappers/python/router_microservice.py b/python/seldon_core/router_microservice.py similarity index 52% rename from wrappers/python/router_microservice.py rename to python/seldon_core/router_microservice.py index 3f8487c1d3..882f9f8e30 100644 --- a/wrappers/python/router_microservice.py +++ b/python/seldon_core/router_microservice.py @@ -1,8 +1,3 @@ -from proto import prediction_pb2, prediction_pb2_grpc -from microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ - array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ - SeldonMicroserviceException, get_custom_tags -from metrics import get_custom_metrics import grpc from concurrent import futures @@ -10,26 +5,38 @@ from flask_cors import CORS import numpy as np import os +import logging + +from seldon_core.proto import prediction_pb2, prediction_pb2_grpc +from seldon_core.microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ + array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ + SeldonMicroserviceException, get_custom_tags +from seldon_core.metrics import get_custom_metrics PRED_UNIT_ID = os.environ.get("PREDICTIVE_UNIT_ID") +logger = logging.getLogger(__name__) + # --------------------------- # Interaction with user router # --------------------------- -def route(user_router,features,feature_names): - return user_router.route(features,feature_names) -def send_feedback(user_router,features,feature_names,routing,reward,truth): - return user_router.send_feedback(features,feature_names,routing,reward,truth) +def route(user_router, features, feature_names): + return user_router.route(features, feature_names) + + +def send_feedback(user_router, features, feature_names, routing, reward, truth): + return user_router.send_feedback(features, feature_names, routing, reward, truth) # ---------------------------- # REST # ---------------------------- -def get_rest_microservice(user_router,debug=False): - app = Flask(__name__,static_url_path='') +def get_rest_microservice(user_router, debug=False): + + app = Flask(__name__, static_url_path='') CORS(app) @app.errorhandler(SeldonMicroserviceException) @@ -38,33 +45,29 @@ def handle_invalid_usage(error): response.status_code = 400 return response - @app.route("/seldon.json",methods=["GET"]) + @app.route("/seldon.json", methods=["GET"]) def openAPI(): - return send_from_directory('', "seldon.json") - + return send_from_directory("openapi", "seldon.json") - @app.route("/route",methods=["GET","POST"]) + @app.route("/route", methods=["GET", "POST"]) def Route(): request = extract_message() - - if debug: - print("SELDON DEBUGGING") - print("Request received: ") - print(request) + logger.debug("Request: %s", request) sanity_check_request(request) datadef = request.get("data") features = rest_datadef_to_array(datadef) - routing = np.array([[route(user_router,features,datadef.get("names"))]]) + routing = np.array( + [[route(user_router, features, datadef.get("names"))]]) # TODO: check that predictions is 2 dimensional class_names = [] data = array_to_rest_datadef(routing, class_names, datadef) - response = {"data":data,"meta":{}} + response = {"data": data, "meta": {}} tags = get_custom_tags(user_router) if tags: response["meta"]["tags"] = tags @@ -73,30 +76,28 @@ def Route(): response["meta"]["metrics"] = metrics return jsonify(response) - @app.route("/send-feedback",methods=["GET","POST"]) + @app.route("/send-feedback", methods=["GET", "POST"]) def SendFeedback(): feedback = extract_message() - if debug: - print("SELDON DEBUGGING") - print("Feedback received: ") - print(feedback) - + logger.debug("Feedback received: %s", feedback) - datadef_request = feedback.get("request",{}).get("data",{}) + datadef_request = feedback.get("request", {}).get("data", {}) features = rest_datadef_to_array(datadef_request) - datadef_truth = feedback.get("truth",{}).get("data",{}) + datadef_truth = feedback.get("truth", {}).get("data", {}) truth = rest_datadef_to_array(datadef_truth) - reward = feedback.get("reward") try: - routing = feedback.get("response").get("meta").get("routing").get(PRED_UNIT_ID) + routing = feedback.get("response").get( + "meta").get("routing").get(PRED_UNIT_ID) except AttributeError: - raise SeldonMicroserviceException("Router feedback must contain a routing dictionary in the response metadata") + raise SeldonMicroserviceException( + "Router feedback must contain a routing dictionary in the response metadata") - send_feedback(user_router,features,datadef_request.get("names"),routing,reward,truth) + send_feedback(user_router, features, datadef_request.get( + "names"), routing, reward, truth) return jsonify({}) return app @@ -107,18 +108,19 @@ def SendFeedback(): # ---------------------------- class SeldonRouterGRPC(object): - def __init__(self,user_model): + def __init__(self, user_model): self.user_model = user_model - def Route(self,request,context): + def Route(self, request, context): datadef = request.data features = grpc_datadef_to_array(datadef) - routing = np.array([[route(self.user_model,features,datadef.names)]]) - #TODO: check that predictions is 2 dimensional + routing = np.array([[route(self.user_model, features, datadef.names)]]) + # TODO: check that predictions is 2 dimensional class_names = [] - data = array_to_grpc_datadef(routing, class_names, request.data.WhichOneof("data_oneof")) + data = array_to_grpc_datadef( + routing, class_names, request.data.WhichOneof("data_oneof")) # Construct meta data meta = prediction_pb2.Meta() @@ -129,11 +131,11 @@ def Route(self,request,context): metrics = get_custom_metrics(self.user_model) if metrics: metaJson["metrics"] = metrics - json_format.ParseDict(metaJson,meta) + json_format.ParseDict(metaJson, meta) - return prediction_pb2.SeldonMessage(data=data,meta=meta) + return prediction_pb2.SeldonMessage(data=data, meta=meta) - def SendFeedback(self,feedback,context): + def SendFeedback(self, feedback, context): datadef_request = feedback.request.data features = grpc_datadef_to_array(datadef_request) @@ -141,19 +143,22 @@ def SendFeedback(self,feedback,context): reward = feedback.reward routing = feedback.response.meta.routing.get(PRED_UNIT_ID) - send_feedback(self.user_model,features,datadef_request.names,routing,reward,truth) + send_feedback(self.user_model, features, + datadef_request.names, routing, reward, truth) return prediction_pb2.SeldonMessage() -def get_grpc_server(user_model,debug=False,annotations={}): + +def get_grpc_server(user_model, debug=False, annotations={}): seldon_router = SeldonRouterGRPC(user_model) options = [] if ANNOTATION_GRPC_MAX_MSG_SIZE in annotations: max_msg = int(annotations[ANNOTATION_GRPC_MAX_MSG_SIZE]) - logger.info("Setting grpc max message to %d",max_msg) - options.append(('grpc.max_message_length', max_msg )) + logger.info("Setting grpc max message to %d", max_msg) + options.append(('grpc.max_message_length', max_msg)) - server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),options=options) + server = grpc.server(futures.ThreadPoolExecutor( + max_workers=10), options=options) prediction_pb2_grpc.add_RouterServicer_to_server(seldon_router, server) return server diff --git a/wrappers/python/seldon_flatbuffers.py b/python/seldon_core/seldon_flatbuffers.py similarity index 60% rename from wrappers/python/seldon_flatbuffers.py rename to python/seldon_core/seldon_flatbuffers.py index 14c40e9ff2..aa67155387 100644 --- a/wrappers/python/seldon_flatbuffers.py +++ b/python/seldon_core/seldon_flatbuffers.py @@ -3,34 +3,38 @@ from tornado import gen import tornado.ioloop -from fbs.SeldonMessage import * -from fbs.Data import * -from fbs.DefaultData import * -from fbs.Tensor import * -from fbs.SeldonRPC import * -from fbs.SeldonPayload import * -from fbs.Status import * -from fbs.StatusValue import * -from fbs.SeldonProtocolVersion import * -from fbs.SeldonRPC import * -from flatbuffers.number_types import (UOffsetTFlags, SOffsetTFlags, VOffsetTFlags) +from flatbuffers.number_types import ( + UOffsetTFlags, SOffsetTFlags, VOffsetTFlags) import sys import numpy as np +from seldon_core.fbs.SeldonMessage import * +from seldon_core.fbs.Data import * +from seldon_core.fbs.DefaultData import * +from seldon_core.fbs.Tensor import * +from seldon_core.fbs.SeldonRPC import * +from seldon_core.fbs.SeldonPayload import * +from seldon_core.fbs.Status import * +from seldon_core.fbs.StatusValue import * +from seldon_core.fbs.SeldonProtocolVersion import * +from seldon_core.fbs.SeldonRPC import * + + class FlatbuffersInvalidMessage(Exception): def __init__(self, msg=None): super(FlatbuffersInvalidMessage, self).__init__(msg) + def SeldonRPCToNumpyArray(data): - seldon_rpc = SeldonRPC.GetRootAsSeldonRPC(data,0) + seldon_rpc = SeldonRPC.GetRootAsSeldonRPC(data, 0) if seldon_rpc.MessageType() == SeldonPayload.SeldonMessage: seldon_msg = SeldonMessage() - seldon_msg.Init(seldon_rpc.Message().Bytes,seldon_rpc.Message().Pos) + seldon_msg.Init(seldon_rpc.Message().Bytes, seldon_rpc.Message().Pos) if seldon_msg.Protocol() == SeldonProtocolVersion.V1: if seldon_msg.DataType() == Data.DefaultData: defData = DefaultData() - defData.Init(seldon_msg.Data().Bytes,seldon_msg.Data().Pos) + defData.Init(seldon_msg.Data().Bytes, seldon_msg.Data().Pos) names = [] for i in range(defData.NamesLength()): names.append(defData.Names(i)) @@ -40,85 +44,86 @@ def SeldonRPCToNumpyArray(data): shape.append(tensor.Shape(i)) values = tensor.ValuesAsNumpy() values = values.reshape(shape) - return (values,names) + return (values, names) else: - raise FlatbuffersInvalidMessage("Message is not of type DefaultData") + raise FlatbuffersInvalidMessage( + "Message is not of type DefaultData") else: - raise FlatbuffersInvalidMessage("Message does not have correct protocol: "+str(seldon_rpc.Protocol())) + raise FlatbuffersInvalidMessage( + "Message does not have correct protocol: " + str(seldon_rpc.Protocol())) else: raise FlatbuffersInvalidMessage("Message is not a SeldonMessage") + def CreateErrorMsg(msg): builder = flatbuffers.Builder(4096) msg_offset = builder.CreateString(msg) - + StatusStart(builder) - StatusAddCode(builder,500) - StatusAddInfo(builder,msg_offset) - StatusAddStatus(builder,StatusValue.FAILURE) + StatusAddCode(builder, 500) + StatusAddInfo(builder, msg_offset) + StatusAddStatus(builder, StatusValue.FAILURE) status = StatusEnd(builder) - + SeldonMessageStart(builder) - SeldonMessageAddStatus(builder,status) + SeldonMessageAddStatus(builder, status) seldonMessage = SeldonMessageEnd(builder) builder.FinishSizePrefixed(seldonMessage) return builder.Output() - - + + # Take a numpy array and create a SeldonRPC message # Creates a local flat buffers builder -def NumpyArrayToSeldonRPC(arr,names): +def NumpyArrayToSeldonRPC(arr, names): builder = flatbuffers.Builder(32768) - if len(names)>0: + if len(names) > 0: str_offsets = [] for i in range(len(names)): str_offsets.append(builder.CreateString(names[i])) - DefaultDataStartNamesVector(builder,len(str_offsets)) + DefaultDataStartNamesVector(builder, len(str_offsets)) for i in reversed(range(len(str_offsets))): builder.PrependUOffsetTRelative(str_offsets[i]) - namesOffset = builder.EndVector(len(str_offsets)) - TensorStartShapeVector(builder,len(arr.shape)) + namesOffset = builder.EndVector(len(str_offsets)) + TensorStartShapeVector(builder, len(arr.shape)) for i in reversed(range(len(arr.shape))): builder.PrependInt32(arr.shape[i]) sOffset = builder.EndVector(len(arr.shape)) arr = arr.flatten() - - #TensorStartValuesVector(builder,len(arr)) - #for i in reversed(range(len(arr))): + + # TensorStartValuesVector(builder,len(arr)) + # for i in reversed(range(len(arr))): # builder.PrependFloat64(arr[i]) #vOffset = builder.EndVector(len(arr)) - vOffset = CreateNumpyVector(builder,arr) - + vOffset = CreateNumpyVector(builder, arr) + TensorStart(builder) - TensorAddShape(builder,sOffset) - TensorAddValues(builder,vOffset) + TensorAddShape(builder, sOffset) + TensorAddValues(builder, vOffset) tensor = TensorEnd(builder) DefaultDataStart(builder) - DefaultDataAddTensor(builder,tensor) - if len(names)>0: - DefaultDataAddNames(builder,namesOffset) + DefaultDataAddTensor(builder, tensor) + if len(names) > 0: + DefaultDataAddNames(builder, namesOffset) defData = DefaultDataEnd(builder) StatusStart(builder) - StatusAddCode(builder,200) - StatusAddStatus(builder,StatusValue.SUCCESS) + StatusAddCode(builder, 200) + StatusAddStatus(builder, StatusValue.SUCCESS) status = StatusEnd(builder) - + SeldonMessageStart(builder) - SeldonMessageAddProtocol(builder,SeldonProtocolVersion.V1) - SeldonMessageAddStatus(builder,status) - SeldonMessageAddDataType(builder,Data.DefaultData) - SeldonMessageAddData(builder,defData) + SeldonMessageAddProtocol(builder, SeldonProtocolVersion.V1) + SeldonMessageAddStatus(builder, status) + SeldonMessageAddDataType(builder, Data.DefaultData) + SeldonMessageAddData(builder, defData) seldonMessage = SeldonMessageEnd(builder) builder.FinishSizePrefixed(seldonMessage) return builder.Output() - - def CreateNumpyVector(builder, x): """CreateNumpyVector writes a numpy array into the buffer.""" @@ -142,12 +147,12 @@ def CreateNumpyVector(builder, x): # Calculate total length l = UOffsetTFlags.py_type(x_lend.itemsize * x_lend.size) - ## @cond FLATBUFFERS_INTERNAL + # @cond FLATBUFFERS_INTERNAL builder.head = UOffsetTFlags.py_type(builder.Head() - l) - ## @endcond + # @endcond # tobytes ensures c_contiguous ordering - builder.Bytes[builder.Head():builder.Head()+l] = x_lend.tobytes(order='C') - - return builder.EndVector(x.size) + builder.Bytes[builder.Head():builder.Head() + + l] = x_lend.tobytes(order='C') + return builder.EndVector(x.size) diff --git a/python/seldon_core/tester.py b/python/seldon_core/tester.py new file mode 100644 index 0000000000..0a9cadbb15 --- /dev/null +++ b/python/seldon_core/tester.py @@ -0,0 +1,303 @@ +import argparse +import numpy as np +import json +import requests +import urllib +from google.protobuf.struct_pb2 import ListValue +import grpc +from time import time + +from seldon_core.proto import prediction_pb2 +from seldon_core.proto import prediction_pb2_grpc + + +def array_to_list_value(array, lv=None): + if lv is None: + lv = ListValue() + if len(array.shape) == 1: + lv.extend(array) + else: + for sub_array in array: + sub_lv = lv.add_list() + array_to_list_value(sub_array, sub_lv) + return lv + + +def gen_continuous(range, n): + if range[0] == "inf" and range[1] == "inf": + return np.random.normal(size=n) + if range[0] == "inf": + return range[1] - np.random.lognormal(size=n) + if range[1] == "inf": + return range[0] + np.random.lognormal(size=n) + return np.random.uniform(range[0], range[1], size=n) + + +def reconciliate_cont_type(feature, dtype): + if dtype == "FLOAT": + return feature + if dtype == "INT": + return (feature + 0.5).astype(int).astype(float) + + +def gen_categorical(values, n): + vals = np.random.randint(len(values), size=n) + return np.array(values)[vals] + + +def generate_batch(contract, n, field): + feature_batches = [] + ty_set = set() + for feature_def in contract[field]: + ty_set.add(feature_def["ftype"]) + if feature_def["ftype"] == "continuous": + if "range" in feature_def: + range = feature_def["range"] + else: + range = ["inf", "inf"] + if "shape" in feature_def: + shape = [n] + feature_def["shape"] + else: + shape = [n, 1] + batch = gen_continuous(range, shape) + batch = np.around(batch, decimals=3) + batch = reconciliate_cont_type(batch, feature_def["dtype"]) + elif feature_def["ftype"] == "categorical": + batch = gen_categorical(feature_def["values"], [n, 1]) + feature_batches.append(batch) + if len(ty_set) == 1: + return np.concatenate(feature_batches, axis=1) + else: + out = np.empty((n, len(contract['features'])), dtype=object) + return np.concatenate(feature_batches, axis=1, out=out) + + +def gen_REST_request(batch, features, tensor=True): + if tensor: + datadef = { + "names": features, + "tensor": { + "shape": batch.shape, + "values": batch.ravel().tolist() + } + } + else: + datadef = { + "names": features, + "ndarray": batch.tolist() + } + + request = { + "meta": {}, + "data": datadef + } + + return request + + +def gen_GRPC_request(batch, features, tensor=True): + if tensor: + datadef = prediction_pb2.DefaultData( + names=features, + tensor=prediction_pb2.Tensor( + shape=batch.shape, + values=batch.ravel().tolist() + ) + ) + else: + datadef = prediction_pb2.DefaultData( + names=features, + ndarray=array_to_list_value(batch) + ) + + request = prediction_pb2.SeldonMessage( + data=datadef + ) + return request + + +def unfold_contract(contract): + unfolded_contract = {} + unfolded_contract["targets"] = [] + unfolded_contract["features"] = [] + + for feature in contract["features"]: + if feature.get("repeat") is not None: + for i in range(feature.get("repeat")): + new_feature = {} + new_feature.update(feature) + new_feature["name"] = feature["name"] + str(i + 1) + del new_feature["repeat"] + unfolded_contract["features"].append(new_feature) + else: + unfolded_contract["features"].append(feature) + + for target in contract["targets"]: + if target.get("repeat") is not None: + for i in range(target.get("repeat")): + new_target = {} + new_target.update(target) + new_target["name"] = target["name"] + ":" + str(i) + del new_target["repeat"] + unfolded_contract["targets"].append(new_target) + else: + unfolded_contract["targets"].append(target) + + return unfolded_contract + + +def run_send_feedback(args): + contract = json.load(open(args.contract, 'r')) + contract = unfold_contract(contract) + feature_names = [feature["name"] for feature in contract["features"]] + response_names = [feature["name"] for feature in contract["targets"]] + + REST_url = "http://" + args.host + ":" + str(args.port) + "/send-feedback" + + for i in range(args.n_requests): + batch = generate_batch(contract, args.batch_size, 'features') + response = generate_batch(contract, args.batch_size, 'targets') + if args.prnt: + print('-' * 40) + print("SENDING NEW REQUEST:") + + if not args.grpc and not args.fbs: + REST_request = gen_REST_request( + batch, features=feature_names, tensor=args.tensor) + REST_response = gen_REST_request( + response, features=response_names, tensor=args.tensor) + reward = 1.0 + REST_feedback = {"request": REST_request, + "response": REST_response, "reward": reward} + if args.prnt: + print(REST_feedback) + + t1 = time() + response = requests.post( + REST_url, + data={"json": json.dumps(REST_feedback)}) + t2 = time() + + if args.prnt: + print("Time " + str(t2 - t1)) + print(response) + elif args.grpc: + GRPC_request = gen_GRPC_request( + batch, features=feature_names, tensor=args.tensor) + GRPC_response = gen_GRPC_request( + response, features=response_names, tensor=args.tensor) + reward = 1.0 + GRPC_feedback = prediction_pb2.Feedback( + request=GRPC_request, + response=GRPC_response, + reward=reward + ) + + if args.prnt: + print(GRPC_feedback) + + channel = grpc.insecure_channel( + '{}:{}'.format(args.host, args.port)) + stub = prediction_pb2_grpc.ModelStub(channel) + response = stub.SendFeedback(GRPC_feedback) + + if args.prnt: + print("RECEIVED RESPONSE:") + print() + + +def run_predict(args): + contract = json.load(open(args.contract, 'r')) + contract = unfold_contract(contract) + feature_names = [feature["name"] for feature in contract["features"]] + + REST_url = "http://" + args.host + ":" + str(args.port) + "/predict" + + for i in range(args.n_requests): + batch = generate_batch(contract, args.batch_size, 'features') + if args.prnt: + print('-' * 40) + print("SENDING NEW REQUEST:") + + if not args.grpc and not args.fbs: + REST_request = gen_REST_request( + batch, features=feature_names, tensor=args.tensor) + if args.prnt: + print(REST_request) + + t1 = time() + response = requests.post( + REST_url, + data={"json": json.dumps(REST_request), "isDefault": True}) + t2 = time() + jresp = response.json() + + if args.prnt: + print("RECEIVED RESPONSE:") + print(jresp) + print() + print("Time " + str(t2 - t1)) + elif args.grpc: + GRPC_request = gen_GRPC_request( + batch, features=feature_names, tensor=args.tensor) + if args.prnt: + print(GRPC_request) + + channel = grpc.insecure_channel( + '{}:{}'.format(args.host, args.port)) + stub = prediction_pb2_grpc.ModelStub(channel) + response = stub.Predict(GRPC_request) + + if args.prnt: + print("RECEIVED RESPONSE:") + print(response) + print() + elif args.fbs: + import socket + import struct + from seldon_core.tester_flatbuffers import NumpyArrayToSeldonRPC, SeldonRPCToNumpyArray + data = NumpyArrayToSeldonRPC(batch, feature_names) + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect((args.host, args.port)) + totalsent = 0 + MSGLEN = len(data) + print("Will send", MSGLEN, "bytes") + while totalsent < MSGLEN: + sent = s.send(data[totalsent:]) + if sent == 0: + raise RuntimeError("socket connection broken") + totalsent = totalsent + sent + data = s.recv(4) + obj = struct.unpack(' 0: + str_offsets = [] + for i in range(len(names)): + str_offsets.append(builder.CreateString(names[i])) + fbs.DefaultData.DefaultDataStartNamesVector(builder, len(str_offsets)) + for i in reversed(range(len(str_offsets))): + builder.PrependUOffsetTRelative(str_offsets[i]) + namesOffset = builder.EndVector(len(str_offsets)) + fbs.Tensor.TensorStartShapeVector(builder, len(arr.shape)) + for i in reversed(range(len(arr.shape))): + builder.PrependInt32(arr.shape[i]) + sOffset = builder.EndVector(len(arr.shape)) + arr = arr.flatten() + fbs.Tensor.TensorStartValuesVector(builder, len(arr)) + for i in reversed(range(len(arr))): + builder.PrependFloat64(arr[i]) + vOffset = builder.EndVector(len(arr)) + fbs.Tensor.TensorStart(builder) + fbs.Tensor.TensorAddShape(builder, sOffset) + fbs.Tensor.TensorAddValues(builder, vOffset) + tensor = fbs.Tensor.TensorEnd(builder) + + fbs.DefaultData.DefaultDataStart(builder) + fbs.DefaultData.DefaultDataAddTensor(builder, tensor) + fbs.DefaultData.DefaultDataAddNames(builder, namesOffset) + defData = fbs.DefaultData.DefaultDataEnd(builder) + + fbs.Status.StatusStart(builder) + fbs.Status.StatusAddCode(builder, 200) + fbs.Status.StatusAddStatus(builder, fbs.StatusValue.StatusValue.SUCCESS) + status = fbs.Status.StatusEnd(builder) + + fbs.SeldonMessage.SeldonMessageStart(builder) + fbs.SeldonMessage.SeldonMessageAddProtocol( + builder, fbs.SeldonProtocolVersion.SeldonProtocolVersion.V1) + fbs.SeldonMessage.SeldonMessageAddStatus(builder, status) + fbs.SeldonMessage.SeldonMessageAddDataType( + builder, fbs.Data.Data.DefaultData) + fbs.SeldonMessage.SeldonMessageAddData(builder, defData) + seldonMessage = fbs.SeldonMessage.SeldonMessageEnd(builder) + + fbs.SeldonRPC.SeldonRPCStart(builder) + fbs.SeldonRPC.SeldonRPCAddMethod( + builder, fbs.SeldonMethod.SeldonMethod.PREDICT) + fbs.SeldonRPC.SeldonRPCAddMessageType( + builder, fbs.SeldonPayload.SeldonPayload.SeldonMessage) + fbs.SeldonRPC.SeldonRPCAddMessage(builder, seldonMessage) + seldonRPC = fbs.SeldonRPC.SeldonRPCEnd(builder) + + builder.FinishSizePrefixed(seldonRPC) + return builder.Output() + + +def SeldonRPCToNumpyArray(data): + seldon_msg = fbs.SeldonMessage.SeldonMessage.GetRootAsSeldonMessage( + data, 0) + if seldon_msg.Protocol() == fbs.SeldonProtocolVersion.SeldonProtocolVersion.V1: + if seldon_msg.DataType() == fbs.Data.Data.DefaultData: + defData = fbs.DefaultData.DefaultData() + defData.Init(seldon_msg.Data().Bytes, seldon_msg.Data().Pos) + names = [] + for i in range(defData.NamesLength()): + names.append(defData.Names(i)) + tensor = defData.Tensor() + shape = [] + for i in range(tensor.ShapeLength()): + shape.append(tensor.Shape(i)) + values = tensor.ValuesAsNumpy() + values = values.reshape(shape) + return (values, names) + else: + raise FlatbuffersInvalidMessage( + "Message is not of type DefaultData") + else: + raise FlatbuffersInvalidMessage( + "Message does not have correct protocol: " + str(seldon_msg.Protocol())) diff --git a/python/seldon_core/transformer_microservice.py b/python/seldon_core/transformer_microservice.py new file mode 100644 index 0000000000..74403fc5b9 --- /dev/null +++ b/python/seldon_core/transformer_microservice.py @@ -0,0 +1,218 @@ +import grpc +from concurrent import futures + +from flask import jsonify, Flask, send_from_directory +from flask_cors import CORS +import numpy as np +from google.protobuf import json_format +import logging + +from seldon_core.proto import prediction_pb2, prediction_pb2_grpc +from seldon_core.microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ + array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ + SeldonMicroserviceException, get_custom_tags, get_data_from_json, get_data_from_proto +from seldon_core.metrics import get_custom_metrics + +logger = logging.getLogger(__name__) + +# --------------------------- +# Interaction with user model +# --------------------------- + + +def transform_input(user_model, features, feature_names): + if hasattr(user_model, "transform_input"): + return user_model.transform_input(features, feature_names) + else: + return features + + +def transform_output(user_model, features, feature_names): + if hasattr(user_model, "transform_output"): + return user_model.transform_output(features, feature_names) + else: + return features + + +def get_feature_names(user_model, original): + if hasattr(user_model, "feature_names"): + return user_model.feature_names + else: + return original + + +def get_class_names(user_model, original): + if hasattr(user_model, "class_names"): + return user_model.class_names + else: + return original + + +# ---------------------------- +# REST +# ---------------------------- + +def get_rest_microservice(user_model, debug=False): + + app = Flask(__name__, static_url_path='') + CORS(app) + + @app.errorhandler(SeldonMicroserviceException) + def handle_invalid_usage(error): + response = jsonify(error.to_dict()) + logger.error("%s", error.to_dict()) + response.status_code = 400 + return response + + @app.route("/seldon.json", methods=["GET"]) + def openAPI(): + return send_from_directory('', "seldon.json") + + @app.route("/transform-input", methods=["GET", "POST"]) + def TransformInput(): + request = extract_message() + logger.debug("Request: %s", request) + + sanity_check_request(request) + + features = get_data_from_json(request) + names = request.get("data", {}).get("names") + + transformed = transform_input(user_model, features, names) + logger.debug("Transformed: %s", transformed) + + # If predictions is an numpy array or we used the default data then return as numpy array + if isinstance(transformed, np.ndarray) or "data" in request: + new_feature_names = get_feature_names(user_model, names) + transformed = np.array(transformed) + data = array_to_rest_datadef( + transformed, new_feature_names, request.get("data", {})) + response = {"data": data, "meta": {}} + else: + response = {"binData": transformed, "meta": {}} + + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) + + @app.route("/transform-output", methods=["GET", "POST"]) + def TransformOutput(): + request = extract_message() + logger.debug("Request: %s", request) + + sanity_check_request(request) + + features = get_data_from_json(request) + names = request.get("data", {}).get("names") + + transformed = transform_output(user_model, features, names) + logger.debug("Transformed: %s", transformed) + + if isinstance(transformed, np.ndarray) or "data" in request: + new_class_names = get_class_names(user_model, names) + data = array_to_rest_datadef( + transformed, new_class_names, request.get("data", {})) + response = {"data": data, "meta": {}} + else: + response = {"binData": transformed, "meta": {}} + + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) + + return app + + +# ---------------------------- +# GRPC +# ---------------------------- + +class SeldonTransformerGRPC(object): + def __init__(self, user_model): + self.user_model = user_model + + def TransformInput(self, request, context): + features = get_data_from_proto(request) + datadef = request.data + data_type = request.WhichOneof("data_oneof") + + transformed = transform_input(self.user_model, features, datadef.names) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson, meta) + + if isinstance(transformed, np.ndarray) or data_type == "data": + transformed = np.array(transformed) + feature_names = get_feature_names(self.user_model, datadef.names) + if data_type == "data": + default_data_type = request.data.WhichOneof("data_oneof") + else: + default_data_type = "tensor" + data = array_to_grpc_datadef( + transformed, feature_names, default_data_type) + return prediction_pb2.SeldonMessage(data=data, meta=meta) + else: + return prediction_pb2.SeldonMessage(binData=transformed, meta=meta) + + def TransformOutput(self, request, context): + features = get_data_from_proto(request) + datadef = request.data + data_type = request.WhichOneof("data_oneof") + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson, meta) + + transformed = transform_output( + self.user_model, features, datadef.names) + + if isinstance(transformed, np.ndarray) or data_type == "data": + transformed = np.array(transformed) + class_names = get_class_names(self.user_model, datadef.names) + if data_type == "data": + default_data_type = request.data.WhichOneof("data_oneof") + else: + default_data_type = "tensor" + data = array_to_grpc_datadef( + transformed, class_names, default_data_type) + return prediction_pb2.SeldonMessage(data=data, meta=meta) + else: + return prediction_pb2.SeldonMessage(binData=transformed, meta=meta) + + +def get_grpc_server(user_model, debug=False, annotations={}): + seldon_model = SeldonTransformerGRPC(user_model) + options = [] + if ANNOTATION_GRPC_MAX_MSG_SIZE in annotations: + max_msg = int(annotations[ANNOTATION_GRPC_MAX_MSG_SIZE]) + logger.info("Setting grpc max message to %d", max_msg) + options.append(('grpc.max_message_length', max_msg)) + + server = grpc.server(futures.ThreadPoolExecutor( + max_workers=10), options=options) + prediction_pb2_grpc.add_ModelServicer_to_server(seldon_model, server) + + return server diff --git a/python/setup.cfg b/python/setup.cfg new file mode 100644 index 0000000000..380e078d2d --- /dev/null +++ b/python/setup.cfg @@ -0,0 +1,6 @@ +[aliases] +test=pytest + +[tool:pytest] +addopts = + --tb native diff --git a/python/setup.py b/python/setup.py new file mode 100644 index 0000000000..85ac80e746 --- /dev/null +++ b/python/setup.py @@ -0,0 +1,39 @@ +from setuptools import find_packages, setup +from seldon_core import __version__ + +setup(name='seldon-core', + author='Seldon Technologies Ltd.', + author_email='hello@seldon.io', + version=__version__, + description='Seldon Core client and microservice wrapper', + url='https://github.com/SeldonIO/seldon-core', + license='Apache 2.0', + packages=find_packages(), + include_package_data=True, + setup_requires=[ + 'pytest-runner' + ], + install_requires=[ + 'flask', + 'flask-cors', + 'redis', + 'tornado', + 'requests', + 'numpy', + 'flatbuffers', + 'protobuf', + 'grpcio', + 'tensorflow' + ], + tests_require=[ + 'pytest', + ], + test_suite='tests', + entry_points={ + 'console_scripts': [ + 'seldon-core-microservice = seldon_core.microservice:main', + 'seldon-core-tester = seldon_core.tester:main', + 'seldon-core-api-tester = seldon_core.api_tester:main', + ], + }, + zip_safe=False) diff --git a/python/tests/model-template-app/.s2i/environment b/python/tests/model-template-app/.s2i/environment new file mode 100644 index 0000000000..dac45a6595 --- /dev/null +++ b/python/tests/model-template-app/.s2i/environment @@ -0,0 +1,4 @@ +MODEL_NAME=MyModel +API_TYPE=REST +SERVICE_TYPE=MODEL +PERSISTENCE=0 diff --git a/python/tests/model-template-app/MyModel.py b/python/tests/model-template-app/MyModel.py new file mode 100644 index 0000000000..8c61bc4dc9 --- /dev/null +++ b/python/tests/model-template-app/MyModel.py @@ -0,0 +1,39 @@ +import logging +logger = logging.getLogger(__name__) + +class MyModel(object): + """ + Model template. You can load your model parameters in __init__ from a location accessible at runtime + """ + + def __init__(self): + """ + Add any initialization parameters. These will be passed at runtime from the graph definition parameters defined in your seldondeployment kubernetes resource manifest. + """ + logger.info("Initializing model") + + def predict(self, X, features_names): + """ + Return a prediction. + + Parameters + ---------- + X : array-like + feature_names : array of feature names (optional) + """ + logger.info("Predict called - will run idenity function") + return X + + def send_feedback(self, features, feature_names, reward, truth): + """ + Handle feedback + + Parameters + ---------- + features : array - the features sent in the original predict request + feature_names : array of feature names. May be None if not available. + reward : float - the reward + truth : array with correct value (optional) + """ + logger.info("Send feedback called") + return [] diff --git a/python/tests/model-template-app/README.md b/python/tests/model-template-app/README.md new file mode 100644 index 0000000000..c2af2473ba --- /dev/null +++ b/python/tests/model-template-app/README.md @@ -0,0 +1,6 @@ + MyModel.py : example template for runtime model + requirements.txt : dependencies + .s2i/environment : add required parameters here + +You can also add a setup.py rather than a requirements.txt + diff --git a/python/tests/model-template-app/contract.json b/python/tests/model-template-app/contract.json new file mode 100644 index 0000000000..6cfca29d8c --- /dev/null +++ b/python/tests/model-template-app/contract.json @@ -0,0 +1,52 @@ +{ + "features":[ + { + "name":"sepal_length", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[ + 4, + 8 + ] + }, + { + "name":"sepal_width", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[ + 2, + 5 + ] + }, + { + "name":"petal_length", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[ + 1, + 10 + ] + }, + { + "name":"petal_width", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[ + 0, + 3 + ] + } + ], + "targets":[ + { + "name":"class", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[ + 0, + 1 + ], + "repeat":3 + } + ] +} diff --git a/python/tests/test_metrics.py b/python/tests/test_metrics.py new file mode 100644 index 0000000000..2aa1817b73 --- /dev/null +++ b/python/tests/test_metrics.py @@ -0,0 +1,115 @@ +import pytest +from google.protobuf import json_format +import json + +from seldon_core.microservice import SeldonMicroserviceException +from seldon_core.proto import prediction_pb2, prediction_pb2_grpc +from seldon_core.metrics import * + + +def test_create_counter(): + v = create_counter("k", 1) + assert v["type"] == "COUNTER" + + +def test_create_counter_invalid_value(): + with pytest.raises(TypeError): + v = create_counter("k", "invalid") + + +def test_create_timer(): + v = create_timer("k", 1) + assert v["type"] == "TIMER" + + +def test_create_timer_invalid_value(): + with pytest.raises(TypeError): + v = create_timer("k", "invalid") + + +def test_create_gauge(): + v = create_gauge("k", 1) + assert v["type"] == "GAUGE" + + +def test_create_gauge_invalid_value(): + with pytest.raises(TypeError): + v = create_gauge("k", "invalid") + + +def test_validate_ok(): + assert validate_metrics( + [{"type": COUNTER, "key": "a", "value": 1}]) == True + + +def test_validate_bad_type(): + assert validate_metrics([{"type": "ABC", "key": "a", "value": 1}]) == False + + +def test_validate_no_type(): + assert validate_metrics([{"key": "a", "value": 1}]) == False + + +def test_validate_no_key(): + assert validate_metrics([{"type": COUNTER, "value": 1}]) == False + + +def test_validate_no_value(): + assert validate_metrics([{"type": COUNTER, "key": "a"}]) == False + + +def test_validate_bad_value(): + assert validate_metrics( + [{"type": COUNTER, "key": "a", "value": "1"}]) == False + + +def test_validate_no_list(): + assert validate_metrics({"type": COUNTER, "key": "a", "value": 1}) == False + + +class Component(object): + + def __init__(self, ok=True): + self.ok = ok + + def metrics(self): + if self.ok: + return [{"type": COUNTER, "key": "a", "value": 1}] + else: + return [{"type": "bad", "key": "a", "value": 1}] + + +def test_component_ok(): + c = Component(True) + assert get_custom_metrics(c) == c.metrics() + + +def test_component_bad(): + with pytest.raises(SeldonMicroserviceException): + c = Component(False) + get_custom_metrics(c) + + +def test_proto_metrics(): + metrics = [{"type": "COUNTER", "key": "a", "value": 1}] + meta = prediction_pb2.Meta() + for metric in metrics: + mpb2 = meta.metrics.add() + json_format.ParseDict(metric, mpb2) + + +def test_proto_tags(): + metric = {"tags": {"t1": "t2"}, "metrics": [{"type": "COUNTER", "key": "mycounter", "value": 1.2}, { + "type": "GAUGE", "key": "mygauge", "value": 1.2}, {"type": "TIMER", "key": "mytimer", "value": 1.2}]} + meta = prediction_pb2.Meta() + json_format.ParseDict(metric, meta) + jStr = json_format.MessageToJson(meta) + j = json.loads(jStr) + assert "mycounter" == j["metrics"][0]["key"] + assert 1.2 == pytest.approx(j["metrics"][0]["value"], 0.01) + assert "GAUGE" == j["metrics"][1]["type"] + assert "mygauge" == j["metrics"][1]["key"] + assert 1.2 == pytest.approx(j["metrics"][1]["value"], 0.01) + assert "TIMER" == j["metrics"][2]["type"] + assert "mytimer" == j["metrics"][2]["key"] + assert 1.2 == pytest.approx(j["metrics"][2]["value"], 0.01) diff --git a/python/tests/test_microservice.py b/python/tests/test_microservice.py new file mode 100644 index 0000000000..589e7f5b77 --- /dev/null +++ b/python/tests/test_microservice.py @@ -0,0 +1,104 @@ +from __future__ import absolute_import, division, print_function + +from contextlib import contextmanager +import json +import os +from os.path import dirname, join +import socket +from subprocess import Popen +import time +import requests + + +@contextmanager +def start_microservice(app_location): + p = None + try: + # PYTHONUNBUFFERED=x + # exec python -u microservice.py $MODEL_NAME $API_TYPE --service-type $SERVICE_TYPE --persistence $PERSISTENCE + env_vars = dict(os.environ) + env_vars.update({ + "PYTHONUNBUFFERED": "x", + "PYTHONPATH": app_location, + "APP_HOST": "127.0.0.1", + "SERVICE_PORT_ENV_NAME": "5000", + }) + with open(join(app_location, ".s2i", "environment")) as fh: + for line in fh.readlines(): + line = line.strip() + if line: + key, value = line.split("=", 1) + key, value = key.strip(), value.strip() + if key and value: + env_vars[key] = value + cmd = ( + "seldon-core-microservice", + env_vars["MODEL_NAME"], + env_vars["API_TYPE"], + "--service-type", env_vars["SERVICE_TYPE"], + "--persistence", env_vars["PERSISTENCE"], + ) + print("starting:", " ".join(cmd)) + print("cwd:", app_location) + # stdout=PIPE, stderr=PIPE, + p = Popen(cmd, cwd=app_location, env=env_vars,) + + for q in range(10): + time.sleep(1) + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + result = sock.connect_ex(("127.0.0.1", 5000)) + if result == 0: + break + else: + raise RuntimeError("Server did not bind to 127.0.0.1:5000") + yield + finally: + if p: + p.terminate() + + +def test_model_template_app(): + with start_microservice(join(dirname(__file__), "model-template-app")): + data = '{"data":{"names":["a","b"],"ndarray":[[1.0,2.0]]}}' + response = requests.get( + "http://127.0.0.1:5000/predict", params="json=%s" % data) + response.raise_for_status() + assert response.json() == { + 'data': {'names': ['t:0', 't:1'], 'ndarray': [[1.0, 2.0]]}, 'meta': {}} + + data = ('{"request":{"data":{"names":["a","b"],"ndarray":[[1.0,2.0]]}},' + '"response":{"meta":{"routing":{"router":0}},"data":{"names":["a","b"],' + '"ndarray":[[1.0,2.0]]}},"reward":1}') + response = requests.get( + "http://127.0.0.1:5000/send-feedback", params="json=%s" % data) + response.raise_for_status() + assert response.json() == {} + + +def test_tester_model_template_app(): + # python api-tester.py contract.json 0.0.0.0 8003 --oauth-key oauth-key --oauth-secret oauth-secret -p --grpc --oauth-port 8002 --endpoint send-feedback + # python tester.py contract.json 0.0.0.0 5000 -p --grpc + with start_microservice(join(dirname(__file__), "model-template-app")): + env_vars = dict(os.environ) + cmd = ( + "seldon-core-tester", + join(dirname(__file__), "model-template-app", "contract.json"), + "127.0.0.1", + "5000", + "--prnt", + ) + print("starting:", " ".join(cmd)) + p = Popen(cmd, env=env_vars,) # stdout=PIPE, stderr=PIPE, + p.wait() + assert p.returncode == 0 + + """ + starting: seldon-core-tester tests/model-template-app/contract.json 127.0.0.1 5000 --prnt + ---------------------------------------- + SENDING NEW REQUEST: + {'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[5.627, 2.239, 9.407, 2.604]]}} + RECEIVED RESPONSE: + {'data': {'names': ['t:0', 't:1', 't:2', 't:3'], 'ndarray': [[5.627, 2.239, 9.407, 2.604]]}} + + Time 0.010219097137451172 + """ diff --git a/python/tests/test_microservice_transformations.py b/python/tests/test_microservice_transformations.py new file mode 100644 index 0000000000..29dffc4801 --- /dev/null +++ b/python/tests/test_microservice_transformations.py @@ -0,0 +1,88 @@ +import pytest +import json +import numpy as np +import pickle +import tensorflow as tf +from google.protobuf import json_format +from tensorflow.core.framework.tensor_pb2 import TensorProto + +from seldon_core.proto import prediction_pb2 +from seldon_core.microservice import get_data_from_json, array_to_grpc_datadef, grpc_datadef_to_array, rest_datadef_to_array, array_to_rest_datadef +from seldon_core.microservice import SeldonMicroserviceException + + +def test_normal_data(): + data = {"data": {"tensor": {"shape": [1, 1], "values": [1]}}} + arr = get_data_from_json(data) + assert isinstance(arr, np.ndarray) + assert arr.shape[0] == 1 + assert arr.shape[1] == 1 + assert arr[0][0] == 1 + + +def test_bin_data(): + a = np.array([1, 2, 3]) + serialized = pickle.dumps(a) + data = {"binData": serialized} + arr = get_data_from_json(data) + assert not isinstance(arr, np.ndarray) + assert arr == serialized + + +def test_str_data(): + data = {"strData": "my string data"} + arr = get_data_from_json(data) + assert not isinstance(arr, np.ndarray) + assert arr == "my string data" + + +def test_bad_data(): + with pytest.raises(SeldonMicroserviceException): + data = {"foo": "bar"} + arr = get_data_from_json(data) + + +def test_proto_array_to_tftensor(): + arr = np.array([[1, 2, 3], [4, 5, 6]]) + datadef = array_to_grpc_datadef(arr, [], "tftensor") + print(datadef) + assert datadef.tftensor.tensor_shape.dim[0].size == 2 + assert datadef.tftensor.tensor_shape.dim[1].size == 3 + assert datadef.tftensor.dtype == 9 + + +def test_proto_tftensor_to_array(): + names = ["a", "b"] + array = np.array([[1, 2], [3, 4]]) + datadef = prediction_pb2.DefaultData( + names=names, + tftensor=tf.make_tensor_proto(array) + ) + array2 = grpc_datadef_to_array(datadef) + assert array.shape == array2.shape + assert np.array_equal(array, array2) + + +def test_json_tftensor_to_array(): + names = ["a", "b"] + array = np.array([[1, 2], [3, 4]]) + datadef = prediction_pb2.DefaultData( + names=names, + tftensor=tf.make_tensor_proto(array) + ) + jStr = json_format.MessageToJson(datadef) + j = json.loads(jStr) + array2 = rest_datadef_to_array(j) + assert np.array_equal(array, array2) + + +def test_json_array_to_tftensor(): + array = np.array([[1, 2], [3, 4]]) + original_datadef = {"tftensor": {}} + datadef = array_to_rest_datadef(array, [], original_datadef) + assert "tftensor" in datadef + tfp = TensorProto() + json_format.ParseDict(datadef.get("tftensor"), tfp, + ignore_unknown_fields=False) + array2 = tf.make_ndarray(tfp) + assert np.array_equal(array, array2) diff --git a/python/tests/test_model_microservice.py b/python/tests/test_model_microservice.py new file mode 100644 index 0000000000..ea5e9feef7 --- /dev/null +++ b/python/tests/test_model_microservice.py @@ -0,0 +1,208 @@ +import pytest +import json +import numpy as np +from google.protobuf import json_format +import base64 +import tensorflow as tf +from tensorflow.core.framework.tensor_pb2 import TensorProto + +from seldon_core.model_microservice import get_rest_microservice, SeldonModelGRPC +from seldon_core.proto import prediction_pb2 + + +class UserObject(object): + def __init__(self, metrics_ok=True, ret_nparray=False): + self.metrics_ok = metrics_ok + self.ret_nparray = ret_nparray + self.nparray = np.array([1, 2, 3]) + + def predict(self, X, features_names): + """ + Return a prediction. + + Parameters + ---------- + X : array-like + feature_names : array of feature names (optional) + """ + if self.ret_nparray: + return self.nparray + else: + print("Predict called - will run identity function") + print(X) + return X + + def tags(self): + return {"mytag": 1} + + def metrics(self): + if self.metrics_ok: + return [{"type": "COUNTER", "key": "mycounter", "value": 1}] + else: + return [{"type": "BAD", "key": "mycounter", "value": 1}] + + +def test_model_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/predict?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_model_tftensor_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + arr = np.array([1, 2]) + datadef = prediction_pb2.DefaultData( + tftensor=tf.make_tensor_proto(arr) + ) + request = prediction_pb2.SeldonMessage(data=datadef) + jStr = json_format.MessageToJson(request) + rv = client.get('/predict?json=' + jStr) + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + assert 'tftensor' in j['data'] + tfp = TensorProto() + json_format.ParseDict(j['data'].get("tftensor"), + tfp, ignore_unknown_fields=False) + arr2 = tf.make_ndarray(tfp) + assert np.array_equal(arr, arr2) + + +def test_model_ok_with_names(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get( + '/predict?json={"data":{"names":["a","b"],"ndarray":[[1,2]]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_model_bin_data(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + bdata = b"123" + bdata_base64 = base64.b64encode(bdata).decode('utf-8') + rv = client.get('/predict?json={"binData":"' + bdata_base64 + '"}') + j = json.loads(rv.data) + sm = prediction_pb2.SeldonMessage() + # Check we can parse response + assert sm == json_format.Parse(rv.data, sm, ignore_unknown_fields=False) + print(j) + assert rv.status_code == 200 + assert j["binData"] == bdata_base64 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_model_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/predict?json={"binData":"123"}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["data"]["ndarray"] == [1, 2, 3] + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_model_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/predict?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_model_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/predict?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_proto_ok(): + user_object = UserObject() + app = SeldonModelGRPC(user_object) + arr = np.array([1, 2]) + datadef = prediction_pb2.DefaultData( + tensor=prediction_pb2.Tensor( + shape=(2, 1), + values=arr + ) + ) + request = prediction_pb2.SeldonMessage(data=datadef) + resp = app.Predict(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["meta"]["tags"] == {"mytag": 1} + # add default type + j["meta"]["metrics"][0]["type"] = "COUNTER" + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["tensor"]["shape"] == [2, 1] + assert j["data"]["tensor"]["values"] == [1, 2] + + +def test_proto_tftensor_ok(): + user_object = UserObject() + app = SeldonModelGRPC(user_object) + arr = np.array([1, 2]) + datadef = prediction_pb2.DefaultData( + tftensor=tf.make_tensor_proto(arr) + ) + request = prediction_pb2.SeldonMessage(data=datadef) + resp = app.Predict(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["meta"]["tags"] == {"mytag": 1} + # add default type + j["meta"]["metrics"][0]["type"] = "COUNTER" + assert j["meta"]["metrics"] == user_object.metrics() + arr2 = tf.make_ndarray(resp.data.tftensor) + assert np.array_equal(arr, arr2) + + +def test_proto_bin_data(): + user_object = UserObject() + app = SeldonModelGRPC(user_object) + bdata = b"123" + bdata_base64 = base64.b64encode(bdata) + request = prediction_pb2.SeldonMessage(binData=bdata_base64) + resp = app.Predict(request, None) + assert resp.binData == bdata_base64 + + +def test_proto_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = SeldonModelGRPC(user_object) + binData = b"\0\1" + request = prediction_pb2.SeldonMessage(binData=binData) + resp = app.Predict(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["data"]["tensor"]["values"] == list(user_object.nparray.flatten()) diff --git a/wrappers/python/test_router_microservice.py b/python/tests/test_router_microservice.py similarity index 61% rename from wrappers/python/test_router_microservice.py rename to python/tests/test_router_microservice.py index 9b3150e2f4..f9d0776669 100644 --- a/wrappers/python/test_router_microservice.py +++ b/python/tests/test_router_microservice.py @@ -1,40 +1,42 @@ import pytest -from router_microservice import get_rest_microservice import json +from seldon_core.router_microservice import get_rest_microservice + + class UserObject(object): - def __init__(self,metrics_ok=True): + def __init__(self, metrics_ok=True): self.metrics_ok = metrics_ok - def route(self,X,features_names): + def route(self, X, features_names): return 22 - + def tags(self): - return {"mytag":1} + return {"mytag": 1} def metrics(self): if self.metrics_ok: - return [{"type":"COUNTER","key":"mycounter","value":1}] + return [{"type": "COUNTER", "key": "mycounter", "value": 1}] else: - return [{"type":"BAD","key":"mycounter","value":1}] - + return [{"type": "BAD", "key": "mycounter", "value": 1}] def test_router_ok(): user_object = UserObject() - app = get_rest_microservice(user_object,debug=True) + app = get_rest_microservice(user_object, debug=True) client = app.test_client() rv = client.get('/route?json={"data":{"ndarray":[2]}}') j = json.loads(rv.data) print(j) assert rv.status_code == 200 - assert j["meta"]["tags"] == {"mytag":1} + assert j["meta"]["tags"] == {"mytag": 1} assert j["meta"]["metrics"] == user_object.metrics() - assert j["data"]["ndarray"] == [[22]] + assert j["data"]["ndarray"] == [[22]] + def test_router_no_json(): user_object = UserObject() - app = get_rest_microservice(user_object,debug=True) + app = get_rest_microservice(user_object, debug=True) client = app.test_client() uo = UserObject() rv = client.get('/route?') @@ -42,12 +44,12 @@ def test_router_no_json(): print(j) assert rv.status_code == 400 + def test_router_bad_metrics(): user_object = UserObject(metrics_ok=False) - app = get_rest_microservice(user_object,debug=True) + app = get_rest_microservice(user_object, debug=True) client = app.test_client() rv = client.get('/route?json={"data":{"ndarray":[]}}') j = json.loads(rv.data) print(j) assert rv.status_code == 400 - diff --git a/python/tests/test_transformer_microservice.py b/python/tests/test_transformer_microservice.py new file mode 100644 index 0000000000..162e4933cb --- /dev/null +++ b/python/tests/test_transformer_microservice.py @@ -0,0 +1,259 @@ +import pytest +import json +import numpy as np +from google.protobuf import json_format +import base64 + +from seldon_core.transformer_microservice import get_rest_microservice, SeldonTransformerGRPC +from seldon_core.proto import prediction_pb2 + + +class UserObject(object): + def __init__(self, metrics_ok=True, ret_nparray=False): + self.metrics_ok = metrics_ok + self.ret_nparray = ret_nparray + self.nparray = np.array([1, 2, 3]) + + def transform_input(self, X, features_names): + if self.ret_nparray: + return self.nparray + else: + print("Transform input called - will run identity function") + print(X) + return X + + def transform_output(self, X, features_names): + if self.ret_nparray: + return self.nparray + else: + print("Transform output called - will run identity function") + print(X) + return X + + def tags(self): + return {"mytag": 1} + + def metrics(self): + if self.metrics_ok: + return [{"type": "COUNTER", "key": "mycounter", "value": 1}] + else: + return [{"type": "BAD", "key": "mycounter", "value": 1}] + + +def test_transformer_input_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-input?json={"data":{"ndarray":[1]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["ndarray"] == [1] + + +def test_transformer_input_bin_data(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + bdata = b"123" + bdata_base64 = base64.b64encode(bdata).decode('utf-8') + rv = client.get('/transform-input?json={"binData":"' + bdata_base64 + '"}') + j = json.loads(rv.data) + sm = prediction_pb2.SeldonMessage() + # Check we can parse response + assert sm == json_format.Parse(rv.data, sm, ignore_unknown_fields=False) + print(j) + assert rv.status_code == 200 + assert "binData" in j + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_transformer_input_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-input?json={"binData":"123"}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["data"]["ndarray"] == [1, 2, 3] + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_tranform_input_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/transform-input?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_transform_input_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-input?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_transformer_output_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-output?json={"data":{"ndarray":[1]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["ndarray"] == [1] + + +def test_transformer_output_bin_data(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + bdata = b"123" + bdata_base64 = base64.b64encode(bdata).decode('utf-8') + rv = client.get( + '/transform-output?json={"binData":"' + bdata_base64 + '"}') + j = json.loads(rv.data) + sm = prediction_pb2.SeldonMessage() + # Check we can parse response + assert sm == json_format.Parse(rv.data, sm, ignore_unknown_fields=False) + print(j) + assert rv.status_code == 200 + assert "binData" in j + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_transformer_output_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-output?json={"binData":"123"}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["data"]["ndarray"] == [1, 2, 3] + assert j["meta"]["tags"] == {"mytag": 1} + assert j["meta"]["metrics"] == user_object.metrics() + + +def test_tranform_output_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/transform-output?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_transform_output_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object, debug=True) + client = app.test_client() + rv = client.get('/transform-output?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + + +def test_transform_input_proto_ok(): + user_object = UserObject() + app = SeldonTransformerGRPC(user_object) + arr = np.array([1, 2]) + datadef = prediction_pb2.DefaultData( + tensor=prediction_pb2.Tensor( + shape=(2, 1), + values=arr + ) + ) + request = prediction_pb2.SeldonMessage(data=datadef) + resp = app.TransformInput(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["meta"]["tags"] == {"mytag": 1} + # add default type + j["meta"]["metrics"][0]["type"] = "COUNTER" + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["tensor"]["shape"] == [2, 1] + assert j["data"]["tensor"]["values"] == [1, 2] + + +def test_transform_input_proto_bin_data(): + user_object = UserObject() + app = SeldonTransformerGRPC(user_object) + binData = b"\0\1" + request = prediction_pb2.SeldonMessage(binData=binData) + resp = app.TransformInput(request, None) + assert resp.binData == binData + + +def test_transform_input_proto_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = SeldonTransformerGRPC(user_object) + binData = b"\0\1" + request = prediction_pb2.SeldonMessage(binData=binData) + resp = app.TransformInput(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["data"]["tensor"]["values"] == list(user_object.nparray.flatten()) + + +def test_transform_output_proto_ok(): + user_object = UserObject() + app = SeldonTransformerGRPC(user_object) + arr = np.array([1, 2]) + datadef = prediction_pb2.DefaultData( + tensor=prediction_pb2.Tensor( + shape=(2, 1), + values=arr + ) + ) + request = prediction_pb2.SeldonMessage(data=datadef) + resp = app.TransformOutput(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["meta"]["tags"] == {"mytag": 1} + # add default type + j["meta"]["metrics"][0]["type"] = "COUNTER" + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["tensor"]["shape"] == [2, 1] + assert j["data"]["tensor"]["values"] == [1, 2] + + +def test_transform_output_proto_bin_data(): + user_object = UserObject() + app = SeldonTransformerGRPC(user_object) + binData = b"\0\1" + request = prediction_pb2.SeldonMessage(binData=binData) + resp = app.TransformOutput(request, None) + assert resp.binData == binData + + +def test_transform_output_proto_bin_data_nparray(): + user_object = UserObject(ret_nparray=True) + app = SeldonTransformerGRPC(user_object) + binData = b"\0\1" + request = prediction_pb2.SeldonMessage(binData=binData) + resp = app.TransformOutput(request, None) + jStr = json_format.MessageToJson(resp) + j = json.loads(jStr) + print(j) + assert j["data"]["tensor"]["values"] == list(user_object.nparray.flatten()) diff --git a/readme.md b/readme.md index 401b0d2276..8960988f6c 100644 --- a/readme.md +++ b/readme.md @@ -129,7 +129,7 @@ Follow the [install guide](docs/install.md) for details on ways to install seldo - [Open API Definitions](./openapi/README.md) - [Seldon Deployment Custom Resource](./docs/reference/seldon-deployment.md) - [Analytics](./docs/analytics.md) - + ## Articles/Blogs/Videos - [Open Source Model Management Roundup Polyaxon, Argo and Seldon](https://www.anaconda.com/blog/developer-blog/open-source-model-management-roundup-polyaxon-argo-and-seldon/) @@ -171,7 +171,7 @@ Follow the [install guide](docs/install.md) for details on ways to install seldo ## Latest Seldon Images -| Description | Image URL | Stable Version | Development | +| Description | Image URL | Stable Version | Development | |-------------|-----------|----------------|-----| | Seldon Operator | [seldonio/cluster-manager](https://hub.docker.com/r/seldonio/cluster-manager/tags/) | 0.2.4 | 0.2.5-SNAPSHOT | | Seldon Service Orchestrator | [seldonio/engine](https://hub.docker.com/r/seldonio/engine/tags/) | 0.2.4 | 0.2.5-SNAPSHOT | @@ -181,13 +181,12 @@ Follow the [install guide](docs/install.md) for details on ways to install seldo | [Seldon Python 3.7 Wrapper for S2I](docs/wrappers/python.md) | [seldonio/seldon-core-s2i-python37](https://hub.docker.com/r/seldonio/seldon-core-s2i-python37/tags/) | 0.3 | 0.4-SNAPSHOT | | [Seldon Python 2 Wrapper for S2I](docs/wrappers/python.md) | [seldonio/seldon-core-s2i-python2](https://hub.docker.com/r/seldonio/seldon-core-s2i-python2/tags/) | 0.3 | 0.4-SNAPSHOT | | [Seldon Python ONNX Wrapper for S2I](docs/wrappers/python.md) | [seldonio/seldon-core-s2i-python3-ngraph-onnx](https://hub.docker.com/r/seldonio/seldon-core-s2i-python3-ngraph-onnx/tags/) | 0.2 | | -| [Seldon Core Python Wrapper](docs/wrappers/python-docker.md) | [seldonio/core-python-wrapper](https://hub.docker.com/r/seldonio/core-python-wrapper/tags/) | 0.7 | | | [Seldon Java Build Wrapper for S2I](docs/wrappers/java.md) | [seldonio/seldon-core-s2i-java-build](https://hub.docker.com/r/seldonio/seldon-core-s2i-java-build/tags/) | 0.1 | | | [Seldon Java Runtime Wrapper for S2I](docs/wrappers/java.md) | [seldonio/seldon-core-s2i-java-runtime](https://hub.docker.com/r/seldonio/seldon-core-s2i-java-runtime/tags/) | 0.1 | | | [Seldon R Wrapper for S2I](docs/wrappers/r.md) | [seldonio/seldon-core-s2i-r](https://hub.docker.com/r/seldonio/seldon-core-s2i-r/tags/) | 0.2 | | | [Seldon NodeJS Wrapper for S2I](docs/wrappers/nodejs.md) | [seldonio/seldon-core-s2i-nodejs](https://hub.docker.com/r/seldonio/seldon-core-s2i-nodejs/tags/) | 0.1 | 0.2-SNAPSHOT | -| [Seldon Tensorflow Serving proxy](integrations/tfserving/README.md) | [seldonio/tfserving-proxy](https://hub.docker.com/r/seldonio/tfserving-proxy/tags/) | 0.1 | -| [Seldon NVIDIA inference server proxy](integrations/nvidia-inference-server/README.md) | [seldonio/nvidia-inference-server-proxy](https://hub.docker.com/r/seldonio/nvidia-inference-server-proxy/tags/) | 0.1 | +| [Seldon Tensorflow Serving proxy](integrations/tfserving/README.md) | [seldonio/tfserving-proxy](https://hub.docker.com/r/seldonio/tfserving-proxy/tags/) | 0.1 | +| [Seldon NVIDIA inference server proxy](integrations/nvidia-inference-server/README.md) | [seldonio/nvidia-inference-server-proxy](https://hub.docker.com/r/seldonio/nvidia-inference-server-proxy/tags/) | 0.1 | #### Java Packages | Description | Package | Version | diff --git a/wrappers-docker/Dockerfile b/wrappers-docker/Dockerfile deleted file mode 100644 index bd8754e908..0000000000 --- a/wrappers-docker/Dockerfile +++ /dev/null @@ -1,36 +0,0 @@ -FROM python:2.7.14-jessie - -COPY _wrappers /wrappers - -# install docker -RUN \ - apt-get update && \ - apt-get install -y \ - apt-transport-https \ - ca-certificates \ - curl \ - gnupg2 \ - software-properties-common && \ - curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | apt-key add - && \ - add-apt-repository \ - "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \ - $(lsb_release -cs) \ - stable" && \ - apt-get update && \ - apt-get install -y docker-ce - - - -RUN python -m pip install grpcio-tools==1.1.3 -RUN cd /wrappers && make build_protos - -# wrapper python dependency -RUN pip install jinja2 - -# deps to get tester.py to work -RUN pip install numpy requests redis flask - -WORKDIR /wrappers/python - -ENTRYPOINT ["python","wrap_model.py"] -CMD [] \ No newline at end of file diff --git a/wrappers-docker/Makefile b/wrappers-docker/Makefile deleted file mode 100644 index 16a2d7f705..0000000000 --- a/wrappers-docker/Makefile +++ /dev/null @@ -1,21 +0,0 @@ -IMAGE_NAME=docker.io/seldonio/core-python-wrapper -IMAGE_VERSION=0.8 - -SELDON_CORE_DIR=.. - -get_wrappers_and_protos: - cp -R $(SELDON_CORE_DIR)/wrappers _wrappers - cp $(SELDON_CORE_DIR)/proto/prediction.proto _wrappers/python/proto - -build_image: clean get_wrappers_and_protos - docker build --force-rm=true -t $(IMAGE_NAME):$(IMAGE_VERSION) . - @docker tag $(IMAGE_NAME):$(IMAGE_VERSION) $(IMAGE_NAME):latest - -push_to_dockerhub: - @ \ - docker push $(IMAGE_NAME):$(IMAGE_VERSION) && \ - docker push $(IMAGE_NAME):latest - -clean: - rm -rfv _wrappers - diff --git a/wrappers-docker/example/Makefile b/wrappers-docker/example/Makefile deleted file mode 100644 index 9ed44c1569..0000000000 --- a/wrappers-docker/example/Makefile +++ /dev/null @@ -1,34 +0,0 @@ -CORE_PYTHON_WAPPER_IMAGE=seldonio/core-python-wrapper:0.3 - -wrap_model: - set -x && docker run --rm -it \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v "$$(pwd):/work" \ - $(CORE_PYTHON_WAPPER_IMAGE) \ - bash -c 'cd /work/models/mean_classifier && make -f makefile.ci wrap_model' - -build_model_image: - set -x && docker run --rm -it \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v "$$(pwd):/work" \ - $(CORE_PYTHON_WAPPER_IMAGE) \ - bash -c 'cd /work/models/mean_classifier && make -f makefile.ci build_model_image' - - -serve_model: - set -x && docker run --rm -it --name=meanclassifier gsunner/meanclassifier:v1_test - -test_model: - set -x && docker run --rm -it --link meanclassifier -v "$$(pwd):/work" $(CORE_PYTHON_WAPPER_IMAGE) bash -c 'python /wrappers/tester.py models/mean_classifier/contract.json $${MEANCLASSIFIER_PORT_5000_TCP_ADDR} $${MEANCLASSIFIER_PORT_5000_TCP_PORT} -p' - -clean: - rm -rfv ./models/mean_classifier/build - - -run_core_python_wrapper: - docker run --rm -it \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v "$$(pwd):/work" \ - $(CORE_PYTHON_WAPPER_IMAGE) \ - bash - diff --git a/wrappers-docker/example/models/mean_classifier/Makefile.ci b/wrappers-docker/example/models/mean_classifier/Makefile.ci deleted file mode 100644 index 819b9924b3..0000000000 --- a/wrappers-docker/example/models/mean_classifier/Makefile.ci +++ /dev/null @@ -1,15 +0,0 @@ -MODEL_NAME=MeanClassifier -MODEL_VERSION=v1_test -MODEL_DOCKER_REPO_NAME=gsunner -BASE_IMAGE=python:2 -MODEL_DIR=/work/models/mean_classifier -MODEL_BUILD_DIR=$(MODEL_DIR)/build -PYTHON_WRAPPERS_DIR=/wrappers/python - -wrap_model: - set -x && rm -rfv $(MODEL_BUILD_DIR) - set -x && cd $(PYTHON_WRAPPERS_DIR) && python wrap_model.py $(MODEL_DIR) $(MODEL_NAME) $(MODEL_VERSION) $(MODEL_DOCKER_REPO_NAME) --base-image $(BASE_IMAGE) && ls -1 $(MODEL_BUILD_DIR) - -build_model_image: - set -x && cd $(MODEL_BUILD_DIR) && make build_docker_image - diff --git a/wrappers-docker/example/models/mean_classifier/MeanClassifier.py b/wrappers-docker/example/models/mean_classifier/MeanClassifier.py deleted file mode 100644 index 42cd333589..0000000000 --- a/wrappers-docker/example/models/mean_classifier/MeanClassifier.py +++ /dev/null @@ -1,29 +0,0 @@ -import numpy as np -import math - -def f(x): - return 1/(1+math.exp(-x)) - -class MeanClassifier(object): - - def __init__(self, intValue=0): - self.class_names = ["proba"] - assert type(intValue) == int, "intValue parameters must be an integer" - self.int_value = intValue - - print "Loading model here" - X = np.load(open("model.npy",'r')) - self.threshold_ = X.mean() + self.int_value - - def _meaning(self, x): - return f(x.mean()-self.threshold_) - - def predict(self, X, feature_names): - print X - X = np.array(X) - assert len(X.shape) == 2, "Incorrect shape" - - return [[self._meaning(x)] for x in X] - - - diff --git a/wrappers-docker/example/models/mean_classifier/contract.json b/wrappers-docker/example/models/mean_classifier/contract.json deleted file mode 100644 index b1e829f44d..0000000000 --- a/wrappers-docker/example/models/mean_classifier/contract.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "features":[ - { - "name":"feat", - "dtype":"FLOAT", - "ftype":"continuous", - "range":["inf","inf"], - "repeat":3 - } - ], - "targets":[ - { - "name":"proba", - "dtype":"FLOAT", - "ftype":"continuous", - "values":[0,1] - } - ] -} diff --git a/wrappers-docker/example/models/mean_classifier/model.npy b/wrappers-docker/example/models/mean_classifier/model.npy deleted file mode 100644 index 4949db57b1..0000000000 Binary files a/wrappers-docker/example/models/mean_classifier/model.npy and /dev/null differ diff --git a/wrappers-docker/example/models/mean_classifier/requirements.txt b/wrappers-docker/example/models/mean_classifier/requirements.txt deleted file mode 100644 index 088485729f..0000000000 --- a/wrappers-docker/example/models/mean_classifier/requirements.txt +++ /dev/null @@ -1,4 +0,0 @@ -numpy==1.11.2 -scikit-learn==0.17.1 -pandas==0.18.1 -scipy==0.18.1 diff --git a/wrappers/python/Dockerfile.tmp b/wrappers/python/Dockerfile.tmp deleted file mode 100644 index 27659c6e1f..0000000000 --- a/wrappers/python/Dockerfile.tmp +++ /dev/null @@ -1,19 +0,0 @@ -FROM {{ base_image }} -{% for key,value in kwargs.iteritems() %} -LABEL "{{ key }}"="{{ value }}"{% endfor %} - -RUN apt-get update -y -RUN apt-get install -y python-pip python-dev build-essential - -COPY /requirements.txt /tmp/ -COPY /seldon_requirements.txt /tmp/ -RUN cd /tmp && \ - pip install --no-cache-dir -r seldon_requirements.txt && \ - pip install --no-cache-dir -r requirements.txt - -RUN mkdir microservice -COPY ./ /microservice/ -WORKDIR /microservice - -EXPOSE 5000 -CMD ["python","-u","microservice.py","{{ model_name }}","{{ api_type }}","--service-type","{{ service_type }}","--persistence","{{ persistence }}"] diff --git a/wrappers/python/README.md.tmp b/wrappers/python/README.md.tmp deleted file mode 100644 index 509c99cccc..0000000000 --- a/wrappers/python/README.md.tmp +++ /dev/null @@ -1,7 +0,0 @@ -Seldon Microservice: {{ model_name }} - -This microservice was wrapped using the Seldon Core Wrappers. - -Wrapping Parameters: -{% for key,value in kwargs.iteritems() %} - {{ key }}: {{ value }} -{% endfor %} \ No newline at end of file diff --git a/wrappers/python/Test.py b/wrappers/python/Test.py deleted file mode 100644 index 4ab817b35f..0000000000 --- a/wrappers/python/Test.py +++ /dev/null @@ -1,48 +0,0 @@ -class Test(object): - """ - Model template. You can load your model parameters in __init__ from a location accessible at runtime - """ - - def __init__(self): - """ - Add any initialization parameters. These will be passed at runtime from the graph definition parameters defined in your seldondeployment kubernetes resource manifest. - """ - print("Initializing") - - def predict(self,X,features_names): - """ - Return a prediction. - - Parameters - ---------- - X : array-like - feature_names : array of feature names (optional) - """ - print("Predict called - will run identity function") - print(X) - return X - - # - # OPTIONAL - # - # This is an optional method that can added to provide custom service endpoints. - # - def custom_service(self): - from flask import Flask, jsonify, request, json - - app = Flask(__name__) - - @app.route("/prometheus_metrics",methods=["GET"]) - def prometheus_metrics(): - return "somemetric 10\n" - - @app.route("/data",methods=["POST"]) - def data(): - data_str = request.data - message = json.loads(data_str) - print(data_str) - return jsonify(message) - - print("Starting custom service") - app.run(host='0.0.0.0', port=5055) - diff --git a/wrappers/python/build_image.sh.tmp b/wrappers/python/build_image.sh.tmp deleted file mode 100644 index 5a7daae062..0000000000 --- a/wrappers/python/build_image.sh.tmp +++ /dev/null @@ -1 +0,0 @@ -docker build --force-rm=true -t {{ docker_repo }}/{{ docker_image_name }}:{{ docker_image_version }} . \ No newline at end of file diff --git a/wrappers/python/fbs/.keep b/wrappers/python/fbs/.keep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/wrappers/python/persistence.py b/wrappers/python/persistence.py deleted file mode 100644 index eedd1ddf90..0000000000 --- a/wrappers/python/persistence.py +++ /dev/null @@ -1,58 +0,0 @@ -import threading -import os -import time -try: - # python 2 - import cPickle as pickle -except ImportError: - # python 3 - import pickle -import redis - - -PRED_UNIT_ID = os.environ.get("PREDICTIVE_UNIT_ID","0") -PREDICTOR_ID = os.environ.get("PREDICTOR_ID","0") -DEPLOYMENT_ID = os.environ.get("SELDON_DEPLOYMENT_ID","0") -REDIS_KEY = "persistence_{}_{}_{}".format(DEPLOYMENT_ID,PREDICTOR_ID,PRED_UNIT_ID) - -REDIS_HOST = os.environ.get('REDIS_SERVICE_HOST','localhost') -REDIS_PORT = os.environ.get("REDIS_SERVICE_PORT",6379) -DEFAULT_PUSH_FREQUENCY = 60 - - -def restore(user_class,parameters,debug=False): - if debug: - print("Restoring saved model from redis") - redis_client = redis.StrictRedis(host=REDIS_HOST,port=REDIS_PORT) - saved_state_binary = redis_client.get(REDIS_KEY) - if saved_state_binary is None: - print("Saved state is empty, restoration aborted") - return user_class(**parameters) - else: - return pickle.loads(saved_state_binary) - -def persist(user_object,push_frequency=None,debug=False): - if push_frequency is None: - push_frequency = DEFAULT_PUSH_FREQUENCY - if debug: - print("Creating persistence thread, with frequency {}".format(push_frequency)) - persistence_thread = PersistenceThread(user_object,push_frequency) - persistence_thread.start() - -class PersistenceThread(threading.Thread): - def __init__(self,user_object,push_frequency): - self.user_object = user_object - self.push_frequency = push_frequency - self._stopped = False - self.redis_client = redis.StrictRedis(host=REDIS_HOST,port=REDIS_PORT) - super(PersistenceThread,self).__init__() - - def stop(self): - print("Stopping Persistence Thread") - self._stopped = True - - def run(self): - while not self._stopped: - time.sleep(self.push_frequency) - binary_data = pickle.dumps(self.user_object) - self.redis_client.set(REDIS_KEY,binary_data) diff --git a/wrappers/python/proto/.keep b/wrappers/python/proto/.keep deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/wrappers/python/push_image.sh.tmp b/wrappers/python/push_image.sh.tmp deleted file mode 100644 index 170fe0f56d..0000000000 --- a/wrappers/python/push_image.sh.tmp +++ /dev/null @@ -1 +0,0 @@ -docker push {{ docker_repo }}/{{ docker_image_name }}:{{ docker_image_version }} diff --git a/wrappers/python/test_metrics.py b/wrappers/python/test_metrics.py deleted file mode 100644 index 3cd56f921f..0000000000 --- a/wrappers/python/test_metrics.py +++ /dev/null @@ -1,97 +0,0 @@ -from metrics import * -import pytest -from microservice import SeldonMicroserviceException -from google.protobuf import json_format -from proto import prediction_pb2, prediction_pb2_grpc -import json - -def test_create_counter(): - v = create_counter("k",1) - assert v["type"] == "COUNTER" - -def test_create_counter_invalid_value(): - with pytest.raises(TypeError): - v = create_counter("k","invalid") - -def test_create_timer(): - v = create_timer("k",1) - assert v["type"] == "TIMER" - -def test_create_timer_invalid_value(): - with pytest.raises(TypeError): - v = create_timer("k","invalid") - -def test_create_gauge(): - v = create_gauge("k",1) - assert v["type"] == "GAUGE" - -def test_create_gauge_invalid_value(): - with pytest.raises(TypeError): - v = create_gauge("k","invalid") - - -def test_validate_ok(): - assert validate_metrics([{"type":COUNTER,"key":"a","value":1}]) == True - -def test_validate_bad_type(): - assert validate_metrics([{"type":"ABC","key":"a","value":1}]) == False - -def test_validate_no_type(): - assert validate_metrics([{"key":"a","value":1}]) == False - -def test_validate_no_key(): - assert validate_metrics([{"type":COUNTER,"value":1}]) == False - -def test_validate_no_value(): - assert validate_metrics([{"type":COUNTER,"key":"a"}]) == False - -def test_validate_bad_value(): - assert validate_metrics([{"type":COUNTER,"key":"a","value":"1"}]) == False - -def test_validate_no_list(): - assert validate_metrics({"type":COUNTER,"key":"a","value":1}) == False - - -class Component(object): - - def __init__(self,ok=True): - self.ok = ok - - def metrics(self): - if self.ok: - return [{"type":COUNTER,"key":"a","value":1}] - else: - return [{"type":"bad","key":"a","value":1}] - - -def test_component_ok(): - c = Component(True) - assert get_custom_metrics(c) == c.metrics() - -def test_component_bad(): - with pytest.raises(SeldonMicroserviceException): - c = Component(False) - get_custom_metrics(c) - -def test_proto_metrics(): - metrics = [{"type":"COUNTER","key":"a","value":1}] - meta = prediction_pb2.Meta() - for metric in metrics: - mpb2 = meta.metrics.add() - json_format.ParseDict(metric,mpb2) - - -def test_proto_tags(): - metric = {"tags":{"t1":"t2"},"metrics":[{"type":"COUNTER","key":"mycounter","value":1.2},{"type":"GAUGE","key":"mygauge","value":1.2},{"type":"TIMER","key":"mytimer","value":1.2}]} - meta = prediction_pb2.Meta() - json_format.ParseDict(metric,meta) - jStr = json_format.MessageToJson(meta) - j = json.loads(jStr) - assert "mycounter" == j["metrics"][0]["key"] - assert 1.2 == pytest.approx(j["metrics"][0]["value"],0.01) - assert "GAUGE" == j["metrics"][1]["type"] - assert "mygauge" == j["metrics"][1]["key"] - assert 1.2 == pytest.approx(j["metrics"][1]["value"],0.01) - assert "TIMER" == j["metrics"][2]["type"] - assert "mytimer" == j["metrics"][2]["key"] - assert 1.2 == pytest.approx(j["metrics"][2]["value"],0.01) diff --git a/wrappers/python/wrap_model.py b/wrappers/python/wrap_model.py deleted file mode 100644 index 50f047e023..0000000000 --- a/wrappers/python/wrap_model.py +++ /dev/null @@ -1,87 +0,0 @@ -import os -import shutil -import argparse -import jinja2 - -def populate_template(filename,build_folder,**kwargs): - with open("./{}.tmp".format(filename),'r') as ftmp: - with open("{}/{}".format(build_folder,filename),'w') as fout: - fout.write(jinja2.Template(ftmp.read()).render(kwargs=kwargs,**kwargs)) - -def wrap_model( - model_folder, - build_folder, - force_erase=False, - **wrapping_arguments): - if os.path.isdir(build_folder): - if not force_erase: - print("Build folder already exists. To force erase, use --force argument") - exit(0) - else: - shutil.rmtree(build_folder) - service_type = wrapping_arguments.get("service_type") - - shutil.copytree(model_folder,build_folder) - shutil.copy2("./Makefile",build_folder) - shutil.copy2('./microservice.py',build_folder) - shutil.copy2("./persistence.py",build_folder) - shutil.copy2('./{}_microservice.py'.format(service_type.lower()),build_folder) - shutil.copy2("./seldon_requirements.txt",build_folder) - shutil.copytree('./proto',build_folder+'/proto') - - populate_template( - 'Dockerfile', - build_folder, - **wrapping_arguments) - populate_template( - "build_image.sh", - build_folder, - **wrapping_arguments) - populate_template( - "push_image.sh", - build_folder, - **wrapping_arguments) - populate_template( - "README.md", - build_folder, - **wrapping_arguments) - - # Make the files executable - st = os.stat(build_folder+"/build_image.sh") - os.chmod(build_folder+"/build_image.sh", st.st_mode | 0111) - st = os.stat(build_folder+"/push_image.sh") - os.chmod(build_folder+"/push_image.sh", st.st_mode | 0111) - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Utility script to wrap a python model into a docker build. The scipt will generate build folder that contains a Makefile that can be used to build and publish a Docker Image.") - - parser.add_argument("model_folder", type=str, help="Path to the folder that contains the model and all the files to copy in the docker image.") - parser.add_argument("model_name", type=str, help="Name of the model class and model file without the .py extension") - parser.add_argument("version", type=str, help="Version string that will be given to the model docker image") - parser.add_argument("repo", type=str, help="Name of the docker repository to publish the image on") - parser.add_argument("--grpc", action="store_true", help="When this flag is present the model will be wrapped as a GRPC microservice. By default the model is wrapped as a REST microservice.") - parser.add_argument("--out-folder", type=str, default=None, help="Path to the folder where the build folder containing the pre-wrapped model will be created. Defaults to the model directory.") - parser.add_argument("--service-type", type=str, choices=["MODEL","ROUTER","TRANSFORMER","COMBINER","OUTLIER_DETECTOR"], default="MODEL", help="The type of Seldon API the wrapped model will use. Defaults to MODEL.") - parser.add_argument("--base-image", type=str, default="python:2", help="The base docker image to inherit from. Defaults to python:2. Caution: this must be a debian based image.") - parser.add_argument("-f", "--force", action="store_true", help="When this flag is present the script will overwrite the contents of the output folder even if it already exists. By default the script would abort.") - parser.add_argument("-p", "--persistence", action="store_true", help="Use redis to make the model persistent") - parser.add_argument("--image-name",type=str,default=None,help="Name to give to the model's docker image. Defaults to the model name in lowercase.") - - args = parser.parse_args() - if args.out_folder is None: - args.out_folder = args.model_folder - if args.image_name is None: - args.image_name = args.model_name.lower() - - wrap_model( - args.model_folder, - args.out_folder+"/build", - force_erase = args.force, - docker_repo = args.repo, - base_image = args.base_image, - model_name = args.model_name, - api_type = "REST" if not args.grpc else "GRPC", - service_type = args.service_type, - docker_image_version = args.version, - docker_image_name = args.image_name, - persistence = int(args.persistence)) diff --git a/wrappers/s2i/python/Dockerfile.local.tmpl b/wrappers/s2i/python/Dockerfile.local.tmpl new file mode 100644 index 0000000000..d7dba5df45 --- /dev/null +++ b/wrappers/s2i/python/Dockerfile.local.tmpl @@ -0,0 +1,18 @@ +FROM python:%PYTHON_VERSION% + +LABEL io.openshift.s2i.scripts-url="image:///s2i/bin" + +RUN apt-get update -y +RUN apt-get install -y python-pip python-dev build-essential + +RUN mkdir microservice +WORKDIR /microservice + +COPY _python /microservice + +RUN cd /microservice/python && \ + pip install . + +COPY ./s2i/bin/ /s2i/bin + +EXPOSE 5000 diff --git a/wrappers/s2i/python/Dockerfile.tmpl b/wrappers/s2i/python/Dockerfile.tmpl index 718610a9c8..c680fdd1fc 100644 --- a/wrappers/s2i/python/Dockerfile.tmpl +++ b/wrappers/s2i/python/Dockerfile.tmpl @@ -8,19 +8,7 @@ RUN apt-get install -y python-pip python-dev build-essential RUN mkdir microservice WORKDIR /microservice -COPY _wrappers/python /microservice - -RUN cd /microservice && \ - pip install --no-cache-dir -r seldon_requirements.txt - -RUN python -m pip install grpcio-tools==1.1.3 -RUN cd /microservice && \ - python -m grpc.tools.protoc -I./ --python_out=./ --grpc_python_out=./ ./proto/prediction.proto - -RUN wget https://github.com/google/flatbuffers/archive/v1.9.0.tar.gz && \ - tar -xvxf v1.9.0.tar.gz && \ - cd flatbuffers-1.9.0/python && \ - python setup.py install +RUN pip install seldon-core COPY ./s2i/bin/ /s2i/bin diff --git a/wrappers/s2i/python/Makefile b/wrappers/s2i/python/Makefile index 42640462ca..39a5e0aee7 100644 --- a/wrappers/s2i/python/Makefile +++ b/wrappers/s2i/python/Makefile @@ -9,31 +9,21 @@ BASE_IMAGE_NAME = docker.io/seldonio/seldon-core-s2i-python${BASE_IMAGE_PYTHON_V SELDON_CORE_DIR=../../.. -.PHONY: get_wrappers_and_protos -get_wrappers_and_protos: - mkdir -p _wrappers/python/proto - mkdir -p _wrappers/python/fbs - cp $(SELDON_CORE_DIR)/wrappers/python/*_microservice.py _wrappers/python - cp $(SELDON_CORE_DIR)/wrappers/python/microservice.py _wrappers/python - cp $(SELDON_CORE_DIR)/wrappers/python/persistence.py _wrappers/python - cp $(SELDON_CORE_DIR)/wrappers/python/seldon_requirements.txt _wrappers/python - cp $(SELDON_CORE_DIR)/wrappers/python/__init__.py _wrappers/python - cd $(SELDON_CORE_DIR)/proto/tensorflow && make create_protos - cp -vr $(SELDON_CORE_DIR)/proto/tensorflow/tensorflow _wrappers/python - cp $(SELDON_CORE_DIR)/proto/prediction.proto _wrappers/python/proto - cp $(SELDON_CORE_DIR)/wrappers/python/seldon_flatbuffers.py _wrappers/python - cp $(SELDON_CORE_DIR)/wrappers/python/metrics.py _wrappers/python - cp -r $(SELDON_CORE_DIR)/proto/tensorflow/tensorflow _wrappers/python/ - flatc --python -o _wrappers/python/fbs ../../../fbs/prediction.fbs - touch _wrappers/python/proto/__init__.py - touch _wrappers/python/fbs/__init__.py - cp $(SELDON_CORE_DIR)/openapi/wrapper.oas3.json _wrappers/python/seldon.json +.PHONY: get_local_repo +get_local_repo: + mkdir -p _python + cp -r $(SELDON_CORE_DIR)/python _python .PHONY: build -build: get_wrappers_and_protos +build: cat Dockerfile.tmpl | sed -e "s|%PYTHON_VERSION%|$(PYTHON_VERSION)|" > Dockerfile set -x && docker build -t $(IMAGE_NAME):$(IMAGE_VERSION) . +.PHONY: build_local +build_local: get_local_repo + cat Dockerfile.local.tmpl | sed -e "s|%PYTHON_VERSION%|$(PYTHON_VERSION)|" > Dockerfile + set -x && docker build -t $(IMAGE_NAME):$(IMAGE_VERSION) . + tag_base_python: docker tag $(IMAGE_NAME):$(IMAGE_VERSION) $(BASE_IMAGE_NAME):$(IMAGE_VERSION) @@ -49,9 +39,15 @@ test: docker build -t $(IMAGE_NAME)-candidate . IMAGE_NAME=$(IMAGE_NAME)-candidate test/run +.PHONY: test_local +test_local: + cat Dockerfile.local.tmpl | sed -e "s|%PYTHON_VERSION%|$(PYTHON_VERSION)|" > Dockerfile + docker build -t $(IMAGE_NAME)-candidate . + IMAGE_NAME=$(IMAGE_NAME)-candidate test/run + .PHONY: clean clean: - rm -rf _wrappers + rm -rf _python rm -rf test/model-template-app/.git rm -rf test/router-template-app/.git rm -rf test/transformer-template-app/.git diff --git a/wrappers/s2i/python/README.md b/wrappers/s2i/python/README.md index e1de157806..41bc321df7 100644 --- a/wrappers/s2i/python/README.md +++ b/wrappers/s2i/python/README.md @@ -7,7 +7,7 @@ e.g. from 0.3-SNAPSHOT to release 0.3 and create 0.4-SNAPSHOT * set IMAGE_VERSION to new stable version X in Makefile (e.g. 0.3) * ```./build_all.sh``` and then ```./push_all.sh``` * Update IMAGE_VERSION to (X+1)-SNAPSHOT (e.g. 0.4-SNAPSHOT) - * ```./build_all.sh``` and then ```./push_all.sh``` + * ```cd build_scripts``` and run ```./build_all.sh``` and then ```./push_all.sh``` * Update main readme to show new versions of stable and snapshot * Update versions in docs, Makefiles and notebooks of stable version ``` ./update_python_version.sh X X+1```, e.g ```./update_python_version.sh 0.2 0.3``` diff --git a/wrappers/s2i/python/build_python2.sh b/wrappers/s2i/python/build_python2.sh deleted file mode 100755 index 350c826e9d..0000000000 --- a/wrappers/s2i/python/build_python2.sh +++ /dev/null @@ -1 +0,0 @@ -make build PYTHON_VERSION=2 diff --git a/wrappers/s2i/python/build_python3.6.sh b/wrappers/s2i/python/build_python3.6.sh deleted file mode 100755 index 93d533276f..0000000000 --- a/wrappers/s2i/python/build_python3.6.sh +++ /dev/null @@ -1,2 +0,0 @@ -make build PYTHON_VERSION=3.6 -make tag_base_python PYTHON_VERSION=3.6 diff --git a/wrappers/s2i/python/build_all.sh b/wrappers/s2i/python/build_scripts/build_all.sh similarity index 100% rename from wrappers/s2i/python/build_all.sh rename to wrappers/s2i/python/build_scripts/build_all.sh diff --git a/wrappers/s2i/python/build_scripts/build_all_local.sh b/wrappers/s2i/python/build_scripts/build_all_local.sh new file mode 100755 index 0000000000..fa5a4b1235 --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_all_local.sh @@ -0,0 +1,3 @@ +./build_local_python2.sh +./build_local_python3.6.sh +./build_local_python3.7.sh diff --git a/wrappers/s2i/python/build_scripts/build_local_python2.sh b/wrappers/s2i/python/build_scripts/build_local_python2.sh new file mode 100755 index 0000000000..fd3b9ab631 --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_local_python2.sh @@ -0,0 +1 @@ +make -C ../ build_local PYTHON_VERSION=2 diff --git a/wrappers/s2i/python/build_scripts/build_local_python3.6.sh b/wrappers/s2i/python/build_scripts/build_local_python3.6.sh new file mode 100755 index 0000000000..31ef63a0db --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_local_python3.6.sh @@ -0,0 +1,2 @@ +make -C ../ build_local PYTHON_VERSION=3.6 +make -C ../ tag_base_python PYTHON_VERSION=3.6 diff --git a/wrappers/s2i/python/build_scripts/build_local_python3.7.sh b/wrappers/s2i/python/build_scripts/build_local_python3.7.sh new file mode 100755 index 0000000000..e2e74fa576 --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_local_python3.7.sh @@ -0,0 +1,3 @@ +# NB: Tensorflow does not work with python 3.7 at present +# see https://github.com/tensorflow/tensorflow/issues/20444 +make -C ../ build_local PYTHON_VERSION=3.7 diff --git a/wrappers/s2i/python/build_scripts/build_python2.sh b/wrappers/s2i/python/build_scripts/build_python2.sh new file mode 100755 index 0000000000..efa3f9740f --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_python2.sh @@ -0,0 +1 @@ +make -C ../ build PYTHON_VERSION=2 diff --git a/wrappers/s2i/python/build_scripts/build_python3.6.sh b/wrappers/s2i/python/build_scripts/build_python3.6.sh new file mode 100755 index 0000000000..de84a264a4 --- /dev/null +++ b/wrappers/s2i/python/build_scripts/build_python3.6.sh @@ -0,0 +1,2 @@ +make -C ../ build PYTHON_VERSION=3.6 +make -C ../ tag_base_python PYTHON_VERSION=3.6 diff --git a/wrappers/s2i/python/build_python3.7.sh b/wrappers/s2i/python/build_scripts/build_python3.7.sh similarity index 76% rename from wrappers/s2i/python/build_python3.7.sh rename to wrappers/s2i/python/build_scripts/build_python3.7.sh index 65f8d365e4..10081ca07a 100755 --- a/wrappers/s2i/python/build_python3.7.sh +++ b/wrappers/s2i/python/build_scripts/build_python3.7.sh @@ -1,3 +1,3 @@ # NB: Tensorflow does not work with python 3.7 at present # see https://github.com/tensorflow/tensorflow/issues/20444 -make build PYTHON_VERSION=3.7 +make -C ../ build PYTHON_VERSION=3.7 diff --git a/wrappers/s2i/python/push_all.sh b/wrappers/s2i/python/build_scripts/push_all.sh similarity index 100% rename from wrappers/s2i/python/push_all.sh rename to wrappers/s2i/python/build_scripts/push_all.sh diff --git a/wrappers/s2i/python/build_scripts/push_python2.sh b/wrappers/s2i/python/build_scripts/push_python2.sh new file mode 100755 index 0000000000..411481f771 --- /dev/null +++ b/wrappers/s2i/python/build_scripts/push_python2.sh @@ -0,0 +1 @@ +make -C ../ push_to_dockerhub diff --git a/wrappers/s2i/python/build_scripts/push_python3.6.sh b/wrappers/s2i/python/build_scripts/push_python3.6.sh new file mode 100755 index 0000000000..13704bca6e --- /dev/null +++ b/wrappers/s2i/python/build_scripts/push_python3.6.sh @@ -0,0 +1,2 @@ +make -C ../ push_to_dockerhub PYTHON_VERSION=3.6 +make -C ../ push_to_dockerhub_base_python PYTHON_VERSION=3.6 diff --git a/wrappers/s2i/python/build_scripts/push_python3.7.sh b/wrappers/s2i/python/build_scripts/push_python3.7.sh new file mode 100755 index 0000000000..c6594feedf --- /dev/null +++ b/wrappers/s2i/python/build_scripts/push_python3.7.sh @@ -0,0 +1 @@ +make -C ../ push_to_dockerhub PYTHON_VERSION=3.7 diff --git a/wrappers/s2i/python/push_python2.sh b/wrappers/s2i/python/push_python2.sh deleted file mode 100755 index 78da56933f..0000000000 --- a/wrappers/s2i/python/push_python2.sh +++ /dev/null @@ -1 +0,0 @@ -make push_to_dockerhub diff --git a/wrappers/s2i/python/push_python3.6.sh b/wrappers/s2i/python/push_python3.6.sh deleted file mode 100755 index 61c382348f..0000000000 --- a/wrappers/s2i/python/push_python3.6.sh +++ /dev/null @@ -1,2 +0,0 @@ -make push_to_dockerhub PYTHON_VERSION=3.6 -make push_to_dockerhub_base_python PYTHON_VERSION=3.6 diff --git a/wrappers/s2i/python/push_python3.7.sh b/wrappers/s2i/python/push_python3.7.sh deleted file mode 100755 index 781363cbe7..0000000000 --- a/wrappers/s2i/python/push_python3.7.sh +++ /dev/null @@ -1 +0,0 @@ -make push_to_dockerhub PYTHON_VERSION=3.7 diff --git a/wrappers/s2i/python/s2i/bin/run b/wrappers/s2i/python/s2i/bin/run index a0fe6af0dc..ebf70bd10d 100755 --- a/wrappers/s2i/python/s2i/bin/run +++ b/wrappers/s2i/python/s2i/bin/run @@ -16,7 +16,7 @@ if [[ -z "$MODEL_NAME" || -z "$API_TYPE" || -z "$SERVICE_TYPE" || -z "$PERSISTEN else cd /microservice echo "starting microservice" - exec python -u microservice.py $MODEL_NAME $API_TYPE --service-type $SERVICE_TYPE --persistence $PERSISTENCE + exec seldon-core-microservice $MODEL_NAME $API_TYPE --service-type $SERVICE_TYPE --persistence $PERSISTENCE fi