Permission denied while reading ./openapi/seldon.json in seldon-container-engine

[aguschin]

Hello, guys. First of all, thank you for developing Seldon-core, its fantastic.

I have tried to access http://<ingress_url>/seldon/<namespace>/<model-name>/api/v1.0/doc/ endpoint to send some requests with UI, as suggested in https://docs.seldon.io/projects/seldon-core/en/latest/workflow/github-readme.html, but stumbled over an error open ./openapi/seldon.json: permission deniedFailed to embed variables on OpenAPI template in seldon-container-engine. Here are steps to reproduce the error (assuming kind is already installed).

To save your time here is the MR which introduced this functionality: #1730

1. kind create cluster

2. installing latest helm chart from repo helm install seldon-core ./helm-charts/seldon-core-operator --namespace seldon-system

3. Using example from https://github.com/SeldonIO/seldon-core/blob/master/executor/README.md

kubectl apply -f - << END
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: seldon-model
spec:
  annotations:
    seldon.io/executor: "true"
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          name: classifier
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    labels:
      version: v1
    name: example
    replicas: 1
END

4. Getting error in logs

k logs seldon-model-example-0-classifier-7fd6fc4d74-v7jc5 -c seldon-container-engine 

{"level":"info","ts":1589971301.715907,"logger":"entrypoint","msg":"Flags","transport":"rest"}
{"level":"info","ts":1589971301.7159576,"logger":"entrypoint","msg":"Trying to get predictor from Env"}
time="2020-05-20T10:41:41Z" level=error msg="open ./openapi/seldon.json: permission deniedFailed to embed variables on OpenAPI template" source="main.go:216"
{"level":"info","ts":1589971301.719009,"logger":"entrypoint","msg":"Starting","worker":1}
{"level":"info","ts":1589971301.7190619,"logger":"entrypoint","msg":"Starting","worker":2}
{"level":"info","ts":1589971301.7190707,"logger":"entrypoint","msg":"Starting","worker":3}
{"level":"info","ts":1589971301.7190747,"logger":"entrypoint","msg":"Starting","worker":4}
{"level":"info","ts":1589971301.7190778,"logger":"entrypoint","msg":"Starting","worker":5}
{"level":"info","ts":1589971301.723948,"logger":"entrypoint","msg":"Running http server ","port":8000}
{"level":"info","ts":1589971301.7246838,"logger":"SeldonRestApi","msg":"Listening","Address":"0.0.0.0:8000"}

5. Finally, when I try to access api/v1.0/doc endpoint, I get "404 page not found". api/v1.0/predictions endpoint works as usual.

That said, this may be a bug or it is me doing something wrong. If this is the case, please comment what steps should be taken to successfully access api/v1.0/doc.

[aguschin]

@axsaucedo, please take a look at this.

[axsaucedo]

@aguschin thanks for reporting I can confirm this can be replicated, we'll have a look why this is the case.

[axsaucedo]

@aguschin, I have just built the executor image locally, which resulted in the correct/expected behaviour without errors. There may be an issue with our automated build. I will have a look why this is the case. However, even if there is an error, if the file exists then you should still be able to access it. You need a trailing slash in doc namely /api/v1.0/doc/, could you try that? This will tell us if the file is there or not. If the file is not there there may be issues with our build pipeline as the image seems to be working correctly.

[aguschin]

Following the above instruction on how to reproduce the error, I did the following:

1. Run k port-forward pod/seldon-model-example-0-classifier-7fd6fc4d74-v7jc5 8000:8000
2. Check that predictions endpoint is working

curl -X POST -H 'Content-Type: application/json'  -d '{ "data": { "ndarray": [[1,2,3]] } }' localhost:8000/api/v1.0/predictions
{"data":{"names":["proba"],"ndarray":[[0.2856375705089441]]},"meta":{}}

3. Check the doc endpoint

curl -X GET -H 'Content-Type: application/json' localhost:8000/api/v1.0/doc
404 page not found

Also I did the same on our dev cluster with usual k8s (not kind installation) with the same results, the only difference is the URL (http://<ingress_url>/seldon/<namespace>/<model-name>/api/v1.0/doc/).

Is this sufficient to answer the question?

Also, in k describe pod/seldon-model-example-0-classifier-7fd6fc4d74-8vzhb, if this will be helpful:

Image:         docker.io/seldonio/seldon-core-executor:1.1.1-SNAPSHOT
Image ID:      docker.io/seldonio/seldon-core-executor@sha256:adda8570f76b8585c6c59ff2675f6606d1bcd5d62a5de0f96783c482dca29caf

[aguschin]

Sorry, I missed out trailing slash. Following the same with port-forward, http://localhost:8000/api/v1.0/doc/ shows the Swagger UI page in browser.

[axsaucedo]

Ok @aguschin I have just confirmed that our build pipeline was having issues, so it has not been building the images correctly. I have triggered a build just now, I will wait until it's finished building and will give you a heads up - it may be enough to fix the issue (as you're currently using the snapshot build).

[aguschin]

I just tried new build which appeared in docker hub, but without luck, same error in logs.

https://hub.docker.com/layers/seldonio/seldon-core-executor/1.1.1-SNAPSHOT/images/sha256-d6a59a0b6f7bd12a151934dd9b7d0ffb1e1c34a87f9f2fcf1d5c654e83babe5d?context=explore

[aguschin]

Hello, @axsaucedo! Any updates on this?