KServe V2 REST Proxy

This REST Proxy leverages gRPC-Gateway to create a reverse-proxy server which translates a RESTful HTTP API into gRPC. This allows sending inference requests using the KServe V2 REST Predict Protocol to platforms that expect the gRPC V2 Predict Protocol.

Note: This is currently a work in progress, and is subject to performance and usability issues.

Install the ProtoBuf compiler

The protocol buffer compiler, protoc is required to compile the .proto files. To install it, follow the instructions here.

Generate the gRPC gateway stubs

After changing the grpc_predict_v2.proto file, run the protoc compiler to regenerate the gRPC gateway code stubs. It's recommended to use the developer image which has all the required libraries pre-installed.

make run generate

Build the Container image

After regenerating the gRPC gateway stubs, rebuild the rest-proxy Docker image.

make build

If you need to use another builder than docker, you can specify it by using the ENGINE variable:

ENGINE=podman make build

Push the Container image

Before pushing the new rest-proxy image to your container registry, re-tag the image created by make build in the step above.

DOCKER_USER="kserve"
DOCKER_TAG="dev"
docker tag kserve/rest-proxy:latest ${DOCKER_USER}/rest-proxy:${DOCKER_TAG}
docker push ${DOCKER_USER}/rest-proxy:${DOCKER_TAG}

Update your ModelMesh deployment

In order to use the newly built rest-proxy image in a ModelMesh Serving deployment update the restProxy.image in config/default/config-defaults.yaml and (re)deploy the ModelMesh Serving.

To update a running deployment of ModelMesh serving, add or update the restProxy.image section in the model-serving-config ConfigMap as described the ModelMesh Serving configuration instructions.