Tfserving proxy custom data

[sebov]

What this PR does / why we need it:
Currently there is no way to handle multiple input/output tensors for the model using Tensorflow Prepackaged Model Server. This PR enables the use of custom_data to pass and receive Tensorflow's predict_pb2.PredictRequest and predict_pb2.PredictResponse respectively. E.g.

import tensorflow as tf
from google.protobuf import any_pb2
from tensorflow_serving.apis import predict_pb2
from tensorflow.core.framework import types_pb2

# an input consisting of multiple ndarrays/tensors
input_data = {
    'input1': input1,
    'input2': input2,
    'input3': input3,
}

# prepare PredictRequest with multiple tensors in 'inputs'
tfrequest = predict_pb2.PredictRequest()
for k, v in input_data.items():
    tfrequest.inputs[k].CopyFrom(tf.make_tensor_proto(input_data[k], types_pb2.DT_FLOAT))

# pack PredictRequest to be acceptable/transferable by Seldon's 'custom_data' argument
custom_data = any_pb2.Any()
custom_data.Pack(tfrequest)

# run grpc prediction, e.g.
grpc_response = SeldonClient().microservice(microservice_endpoint='localhost:9500',
                                     method='predict',
                                     transport='grpc',
                                     custom_data=custom_data)

# unpack the response to PredictResponse
tfresponse = predict_pb2.PredictResponse()
grpc_response.response.customData.Unpack(tfresponse)

# translate outputs back to ndarrays
result = {} # Dict[str, np.ndarray]
result['output1'] = tf.make_ndarray(tfresponse.outputs['output1'])
result['output2'] = tf.make_ndarray(tfresponse.outputs['output2'])

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Added custom_data handling to Tensorflow Prepackaged Model Server

[seldondev]

Hi @sebov. Thanks for your PR.

I'm waiting for a SeldonIO or todo member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[seldondev]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign sachinvarghese
You can assign the PR to them by writing /assign @sachinvarghese in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sebov]

/assign @SachinVarghese

[ukclivecox]

/ok-to-test

[ukclivecox]

/test all

[sebov]

/test release

[seldondev]

@sebov: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
release-build-push	d35d48c	link	/test release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

[sebov]

Hi @cliveseldon. May I ask what happened in this PR? It would be a great help for my future contributions. I can see that one of the test "release-build-push" failed for commit "d35d48c". I cannot see the details (there is a login popup when clicked on the link), however this particular commit was concerned only the code reformatting by "black" code formatter - like described on your site https://docs.seldon.io/projects/seldon-core/en/v1.1.0/developer/contributing.html and in .pre-commit-config.yaml.