connect timed out with cluster-manager and api server

[kaysonx]

I followed the instruction with helm to install the seldon core, but get follow error:

Unexpected error trying to create CRD with:
io.kubernetes.client.ApiException: java.net.SocketTimeoutException: connect timed out

Failed to instantiate [io.seldon.clustermanager.k8s.SeldonDeploymentWatcher]: Constructor threw exception; nested exception is io.kubernetes.client.ApiException: java.net.SocketTimeoutException: connect timed out

and I've checked the role/rolebinding/sa, it's the excepted with helm charts.

Any suggestions folks?

BTW, the command I use is:

helm install ./seldon-core-crd-0.2.6.tgz  --name seldon-core-crd  --set usage_metrics.enabled=true

helm install ./seldon-core-0.2.6.tgz  --name seldon-core  --namespace seldon \
--set apife.image.name=my-private-registry/apife:0.2.6 \
--set cluster_manager.image.name=my-private-registry/cluster-manager:0.2.6 \
--set engine.image.name=my-private-registry/engine:0.2.6 \
--set redis.image.name=my-private-registry/redis:4.0.1

[ukclivecox]

This seems to suggest the cluster-manager pod can't connect to the k8s API. Is there anything special about the cluster you are running on? Does it allow k8s API access from your namespace or does the RBAC of your cluster disallow this?

[kaysonx]

Thanks your reply!

The cluster is ok, we already have a customer scheduler with client-go sdk writing and running on it.
Just check the pod, the token is mount rightly with:
/var/run/secrets/kubernetes.io/serviceaccount from seldon-token-mwcpb

and the corresponding role is:

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    creationTimestamp: 2019-04-24T09:34:00Z
    name: seldon-local
    namespace: seldon
    resourceVersion: "40929244"
    selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/seldon/roles/seldon-local
    uid: 176d39f6-6674-11e9-a41d-70106fba9de6
  rules:
  - apiGroups:
    - '*'
    resources:
    - deployments
    - services
    verbs:
    - '*'
  - apiGroups:
    - machinelearning.seldon.io
    resources:
    - '*'
    verbs:
    - '*'
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I guess the k8s java sdk using this mount token: /var/run/secrets/kubernetes.io/serviceaccount
Is that right? or the role need more permission with.

[ukclivecox]

Yes it will be using seldon-local RBAC and the default token that k8s adds to the pod. Is your cluster setup with any restrictions?

[ukclivecox]

Please reopen if still an issue on 0.4.0 release.