Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issues on increasing grpc message size #467

Closed
sathiez opened this issue Mar 20, 2019 · 15 comments
Closed

issues on increasing grpc message size #467

sathiez opened this issue Mar 20, 2019 · 15 comments
Assignees

Comments

@sathiez
Copy link

sathiez commented Mar 20, 2019

when I send request with size which exceeds default 4mb , I get error eventhough I configured for grpc max message size link.
Below is my seldon image log

starting microservice 2019-03-20 06:42:34,493 - seldon_core.microservice:main:261 - INFO: Starting microservice.py:main 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation deployment_version:v1 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation kubernetes.io/config.seen:2019-03-20T12:12:31.811630536+05:30 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation kubernetes.io/config.source:api 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation predictor_version:v1 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation project_name:crop-face 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation seldon.io/grpc-max-message-size:10485760 2019-03-20 06:42:34,494 - seldon_core.microservice:main:292 - INFO: Annotations: {'deployment_version': 'v1', 'kubernetes.io/config.seen': '2019-03-20T12:12:31.811630536+05:30', 'kubernetes.io/config.source': 'api', 'predictor_version': 'v1', 'project_name': 'crop-face', 'seldon.io/grpc-max-message-size': '10485760'} 2019-03-20 06:42:35,929 - seldon_core.microservice:main:354 - INFO: Starting servers loading... 2019-03-20 06:42:35,934 - seldon_core.model_microservice:get_grpc_server:197 - INFO: Setting grpc max message and receive length to 10485760 2019-03-20 06:42:35,943 - seldon_core.microservice:grpc_prediction_server:333 - INFO: GRPC microservice Running on port 9000

My error is

raise _Rendezvous(state, None, None, deadline) grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.CANCELLED details = "Received RST_STREAM with error code 8" debug_error_string = "{"created":"@1553065266.933206950","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 8","grpc_status":1}"

SVC-orch log is

2019-03-20 07:01:06.936 WARN 8 --- [worker-ELG-3-14] io.grpc.netty.NettyServerHandler : Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 1 at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final]

How can I solve it?

@ukclivecox
Copy link
Contributor

The log from the svc-orch Received DATA frame for an unknown stream 1 seems strange.

Can you check this example notebook works for you: https://docs.seldon.io/projects/seldon-core/en/latest/graph/model_rest_grpc_settings.html

Can you add the other annotations for increasing Timeouts in case its also connected to that?

@ukclivecox
Copy link
Contributor

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

I increased the time out as mentioned in that notebook, but still I am unable to crack it.., :(

2019-03-20 08:20:38,109 - seldon_core.microservice:main:292 - INFO: Annotations: {'deployment_version': 'v1', 'kubernetes.io/config.seen': '2019-03-20T13:50:35.385232702+05:30', 'kubernetes.io/config.source': 'api', 'predictor_version': 'v1', 'project_name': 'crop-face', 'seldon.io/grpc-max-message-size': '10000000', 'seldon.io/grpc-read-timeout': '10000000', 'seldon.io/rest-connection-timeout': '10000000', 'seldon.io/rest-read-timeout': '10000000'}

@ukclivecox
Copy link
Contributor

Can you run the notebook to confirm it works for you.

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

That example works fine but I am unable to get result in my model.
My code is,

namespace = "kubeflow" deploymentname = "crop-face" stub = prediction_pb2_grpc.SeldonStub(channel) request = prediction_pb2.SeldonMessage(binData = img.tobytes()) metadata = [('seldon',deploymentname),('namespace',namespace)] response = stub.Predict(request=request,metadata=metadata)

any troubleshooting method?

@ukclivecox
Copy link
Contributor

Can you install the latest python module seldon-core and use our reference python client to call:

https://docs.seldon.io/projects/seldon-core/en/latest/python/python_module.html#seldon-core-python-api-client

There is an example notebook referenced on that page.

You should be able to send binData:

https://docs.seldon.io/projects/seldon-core/en/latest/python/api/seldon_core.html#seldon_core.seldon_client.SeldonClient.predict

@ukclivecox
Copy link
Contributor

Can you send a small image using that endpoint as it may not be to do with the grpc message size but something more basic.

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

I tried with small image, it is working fine... but when I increase image size I get error.

@ukclivecox
Copy link
Contributor

Can you check the svc-orch prints out log messages that it also is setting max grpc message size?
Which version of Seldon Core are you running with?

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

you are right, svc-orph max-message size is "gRPC max message size set to 4194304" .
my seldon version = seldonio:0.2.3

@ukclivecox
Copy link
Contributor

If you are able to use a more recent version of Seldon that would be useful. You could try installing Seldon via our Helm charts : https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

Hi I am using kubeflow/seldon how to update it with ksonnet??? any suggestions please...

@ukclivecox
Copy link
Contributor

You could use the ksonnet repo as described in the docs above rather than the kubeflow seldon ksonnet package

@sathiez
Copy link
Author

sathiez commented Mar 20, 2019

Hi, I successfully installed seldon using ksonnet, my seldon version is now "seldonio/cluster-manager:0.2.7-SNAPSHOT" but after setting grpc max message size limit the issue remains the same, I can successfully process image less than 4 mb but if it crossed I get the below grpc error:

status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (5340013 vs. 4194304)" debug_error_string = "{"created":"@1553081440.878999727","description":"Received message larger than max (5340013 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}"

and my seldon logs :

2019-03-20 11:28:39,552 - seldon_core.microservice:main:354 - INFO: Starting servers loading... 2019-03-20 11:28:39,559 - seldon_core.model_microservice:get_grpc_server:197 - INFO: Setting grpc max message and receive length to 10485760 2019-03-20 11:28:39,567 - seldon_core.microservice:grpc_prediction_server:333 - INFO: GRPC microservice Running on port 9000

@ukclivecox
Copy link
Contributor

Assume fixed now. Please reopen if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants