-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
issues on increasing grpc message size #467
Comments
The log from the svc-orch Can you check this example notebook works for you: https://docs.seldon.io/projects/seldon-core/en/latest/graph/model_rest_grpc_settings.html Can you add the other annotations for increasing Timeouts in case its also connected to that? |
I meant to reference this notebook: https://docs.seldon.io/projects/seldon-core/en/latest/examples/max_grpc_msg_size.html |
I increased the time out as mentioned in that notebook, but still I am unable to crack it.., :(
|
Can you run the notebook to confirm it works for you. |
That example works fine but I am unable to get result in my model.
any troubleshooting method? |
Can you install the latest python module There is an example notebook referenced on that page. You should be able to send binData: |
Can you send a small image using that endpoint as it may not be to do with the grpc message size but something more basic. |
I tried with small image, it is working fine... but when I increase image size I get error. |
Can you check the svc-orch prints out log messages that it also is setting max grpc message size? |
you are right, svc-orph max-message size is "gRPC max message size set to 4194304" . |
If you are able to use a more recent version of Seldon that would be useful. You could try installing Seldon via our Helm charts : https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html |
Hi I am using kubeflow/seldon how to update it with ksonnet??? any suggestions please... |
You could use the ksonnet repo as described in the docs above rather than the kubeflow seldon ksonnet package |
Hi, I successfully installed seldon using ksonnet, my seldon version is now "seldonio/cluster-manager:0.2.7-SNAPSHOT" but after setting grpc max message size limit the issue remains the same, I can successfully process image less than 4 mb but if it crossed I get the below grpc error:
and my seldon logs :
|
Assume fixed now. Please reopen if not. |
when I send request with size which exceeds default 4mb , I get error eventhough I configured for grpc max message size link.
Below is my seldon image log
starting microservice 2019-03-20 06:42:34,493 - seldon_core.microservice:main:261 - INFO: Starting microservice.py:main 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation deployment_version:v1 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation kubernetes.io/config.seen:2019-03-20T12:12:31.811630536+05:30 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation kubernetes.io/config.source:api 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation predictor_version:v1 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation project_name:crop-face 2019-03-20 06:42:34,494 - seldon_core.microservice:load_annotations:249 - INFO: Found annotation seldon.io/grpc-max-message-size:10485760 2019-03-20 06:42:34,494 - seldon_core.microservice:main:292 - INFO: Annotations: {'deployment_version': 'v1', 'kubernetes.io/config.seen': '2019-03-20T12:12:31.811630536+05:30', 'kubernetes.io/config.source': 'api', 'predictor_version': 'v1', 'project_name': 'crop-face', 'seldon.io/grpc-max-message-size': '10485760'} 2019-03-20 06:42:35,929 - seldon_core.microservice:main:354 - INFO: Starting servers loading... 2019-03-20 06:42:35,934 - seldon_core.model_microservice:get_grpc_server:197 - INFO: Setting grpc max message and receive length to 10485760 2019-03-20 06:42:35,943 - seldon_core.microservice:grpc_prediction_server:333 - INFO: GRPC microservice Running on port 9000
My error is
raise _Rendezvous(state, None, None, deadline) grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.CANCELLED details = "Received RST_STREAM with error code 8" debug_error_string = "{"created":"@1553065266.933206950","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Received RST_STREAM with error code 8","grpc_status":1}"
SVC-orch log is
2019-03-20 07:01:06.936 WARN 8 --- [worker-ELG-3-14] io.grpc.netty.NettyServerHandler : Stream Error io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 1 at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183) ~[netty-codec-http2-4.1.27.Final.jar!/:4.1.27.Final]
How can I solve it?
The text was updated successfully, but these errors were encountered: