How do I increase timeout of sidecar seldon container?

[codebreach]

We see the following error as one of our predictors takes a while (10+ sec)

But it seems that the sidecar that manages these connections has a fixed timeout in the code somewhere. Ideally we should allow this to be configured somewhere?

{
    "code": 203,
    "info": "org.springframework.web.client.ResourceAccessException: I/O error on POST request for \"http://0.0.0.0:9000/predict\": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out",
    "reason": "Microservice error",
    "status": "FAILURE"
}

[codebreach]

Looking at the code

It seems that its hardcoded

seldon-core/api-frontend/src/main/java/io/seldon/apife/service/InternalPredictionService.java

Line 55 in 93d875d

private static final int DEFAULT_SOCKET_TIMEOUT = 2000;

[ukclivecox]

Yes, you are right ; we are working on functionality to allow configuration via annotations in the SeldonDeployment CRD so these things can be configurable.

[codebreach]

Is there an PR that has the code? we can dogfood it because its preventing us from deploying to production.

we can also help implement if required.

[codebreach]

also @cliveseldon would it be possible for us to fork the repo and update the hardcoded value and use that...like can we tell seldon to use a different image/tag for the sidecar?

that way we can do a quick and dirty change while you guys make the required changes to the CRD etc.

[codebreach]

seems that if we can somehow set the property
io.seldon.clustermanager.engine-container-image-and-version=${ENGINE_CONTAINER_IMAGE_AND_VERSION}

it will work.

any guidance?

[ukclivecox]

Yes, there are a few things:

The timeout will be in the API FrontEnd Gateway AND the Seldon Engine (service orchestrator), so you would need to hack both of those. Yes, you can change the engine version from the Helm values.yaml.

seldon-core/helm-charts/seldon-core/values.yaml

Line 33 in 8b6bf70

name: seldonio/engine:0.2.3-SNAPSHOT

I will do a premlinary PR with the code for the annotations as well. Happy to have help on working on this.

Please contact me on our Slack channel.

[ukclivecox]

The PR #197

[ukclivecox]

This should be fixed in the latest snapshot development release by the PR #211
If you use images for 0.2.3-SNAPSHOT you can test. See
docs and
example.

Please test and tell us if it fixes your requirements.

[codebreach]

This is great! Will test it out and let you know. It seems that this should cover our use-case exactly. Madhav

…

On Thu, Aug 30, 2018 at 4:08 PM cliveseldon ***@***.***> wrote: This should be fixed in the latest snapshot development release by the PR #211 <#211> If you use images for 0.2.3-SNAPSHOT you can test. See docs <https://github.com/SeldonIO/seldon-core/blob/master/docs/annotations.md> and example <https://github.com/SeldonIO/seldon-core/blob/master/notebooks/timeouts.ipynb> . Please test and tell us if it fixes your requirements. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#196 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAenDPFvz1hx_x1xRo_RY2R6Xtai_BJ0ks5uV8CYgaJpZM4VqXhE> .

[codebreach]

This doesn't work behind ambassador
timeout_ms is not set on the service generated even though we pass the annotations

kind: Service
metadata:
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  seldon_dpa_rest_mapping
      prefix: /seldon/dpa/
      service: dpa:8000
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  dpa_grpc_mapping
      grpc: true
      prefix: /seldon.protos.Seldon/
      rewrite: /seldon.protos.Seldon/
      headers:
        seldon: dpa
      service: dpa:5001

it should include

      timeout_ms: VALUE_FROM_ANNOTATION

[codebreach]

please let me know if i should open a new issue

[ukclivecox]

Can you add your case to #120