Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sklearn iris returns value error #389

Closed
vincentclaes opened this issue Jan 15, 2019 · 11 comments
Closed

sklearn iris returns value error #389

vincentclaes opened this issue Jan 15, 2019 · 11 comments

Comments

@vincentclaes
Copy link

I am following the tutorial sklearn iris tutorial but there goes something wrong when returning the prediction

tutorial:

https://github.com/SeldonIO/seldon-core/blob/master/examples/models/sklearn_iris/sklearn_iris.ipynb

command / output:

seldon-core-tester contract.json 0.0.0.0 5000 -p

SENDING NEW REQUEST:
{'meta': {}, 'data': {'names': [u'sepal_length', u'sepal_width', u'petal_length', u'petal_width'], 'ndarray': [[5.706, 4.111, 8.179, 1.42]]}}
Traceback (most recent call last):
File "/usr/local/bin/seldon-core-tester", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/seldon_core/tester.py", line 297, in main
run_predict(args)
File "/usr/local/lib/python2.7/dist-packages/seldon_core/tester.py", line 233, in run_predict
jresp = response.json()
File "/home/dvcl643/.local/lib/python2.7/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

@jklaise
Copy link
Contributor

jklaise commented Jan 15, 2019

@vincentclaes I can't replicate the issue, I've tested both Python 2 and 3 environments and the tester code works as expected. Did you follow the notebook and wrap your own image using s2i adn the seldonio/seldon-core-s2i-python3:0.4 builder image or did you make any changes?

@vincentclaes
Copy link
Author

@vincentclaes I can't replicate the issue, I've tested both Python 2 and 3 environments and the tester code works as expected. Did you follow the notebook and wrap your own image using s2i adn the seldonio/seldon-core-s2i-python3:0.4 builder image or did you make any changes?

Hi jklaise,

what i do is;

  • i go to seldon-core/examples/models/sklearn_iris
  • pipenv install --python 3.6 (to create virtualenv and install deps)
  • pipenv shell (start virtual env)
  • run python IrisClassifier.py (using python 3.6.8) to create the .sav file

i get the first error here:
Traceback (most recent call last):
File "IrisClassifier.py", line 1, in
from sklearn.externals import joblib
File "/home/dvcl643/.local/share/virtualenvs/sklearn_iris-F1tHn4vx/lib/python3.6/site-packages/sklearn/init.py", line 134, in
from .base import clone
File "/home/dvcl643/.local/share/virtualenvs/sklearn_iris-F1tHn4vx/lib/python3.6/site-packages/sklearn/base.py", line 9, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'

@jklaise
Copy link
Contributor

jklaise commented Jan 15, 2019

This means that numpy is missing as a dependency. Since it should be automatically pulled in by sklearn we haven't added it to the requirements.txt, but I see you are using pipenv to create virtual environments which manages dependencies in a different way to pip/conda, so this could be the issue?

Also, the IrisClassifier.py contains the model class that will load the saved model weights (.sav) after training and shouldn't be called directly, you should run the model training code provided in the notebook and then use s2i to build the docker image from IrisClassifier.py. When you work through the notebook does everything else up to the first seldon-core-tester call work as expected (training, wrapping and running the docker container)?

@vincentclaes
Copy link
Author

vincentclaes commented Jan 15, 2019

@jklaise sorry i had to run the train_iris.py file instead of the IrisClassifier.py. i was confused ... so if i run the train_iris file i get a IrisClassifier.sav -> ok

if i follow the jupyter notebook, everything runs fine up until the seldon-core-tester.
if i run the notebook i get a similar issue I had earlier today. It looks like it has to do with my seldon-core(-tester) client application if i look at the stacktrace i get:

SENDING NEW REQUEST:
{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[7.958, 3.851, 8.423, 2.17]]}}
Traceback (most recent call last):
  File "/usr/local/bin/seldon-core-tester", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 297, in main
    run_predict(args)
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 233, in run_predict
    jresp = response.json()
  File "/usr/local/lib/python3.6/site-packages/requests-2.21.0-py3.6.egg/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

i use the following version:
seldon-core==0.2.5

image

@jklaise
Copy link
Contributor

jklaise commented Jan 16, 2019

@vincentclaes this is rather strange, the s2i build succeeds and it looks like the docker container is running too (just to confirm, does it show up on docker container ls?). One thing to try is to check if the seldon-core-tester script works with the pre-built image, i.e. if you run docker run --name "iris_predictor" -d --rm -p 5000:5000 seldonio/sklearn-iris:0.1 to download and run the pre-built image from dockerhub. If the tester script works on this image we know there is an issue in the s2i build, if it doesn't then it might be an issue with the script or your environment.

@vincentclaes
Copy link
Author

vincentclaes commented Jan 16, 2019

the pre built image runs:

$ docker container ls
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS                    NAMES
118f130f6d70        seldonio/sklearn-iris:0.1   "/s2i/bin/run"      13 minutes ago      Up 13 minutes       0.0.0.0:5000->5000/tcp   iris_predictor

with the pre-built image i get the same output


$ seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:
{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[4.377, 4.051, 7.61, 0.01]]}}
Traceback (most recent call last):
  File "/usr/local/bin/seldon-core-tester", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 297, in main
    run_predict(args)
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 233, in run_predict
    jresp = response.json()
  File "/usr/local/lib/python3.6/site-packages/requests-2.21.0-py3.6.egg/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I see that there is no older version of seldon-core available on pypi
i tested with python 2.7.12 and python 3.6.8
os i am using (actually windows subsystem linux for windows10):

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

i tested the script locally and it returns an output (a numpy array)

@jklaise
Copy link
Contributor

jklaise commented Jan 16, 2019

Can you post the output of docker logs iris_predictor? It just occurred to me that if you're running the training script with a newer version of scikit-learn and then using the version in the requirements.txt to read in the pickled model it might cause compatibility issues.

@vincentclaes
Copy link
Author

so in short i have scikit-learn == 0.19.0 the same as in the requirements.
i have build the container again, ran seldon-core-tester and got the usual error the logs from my docker does not mention anything special.

dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ s2i build . seldonio/seldon-core-s2i-python3:0.4 sklearn-iris:0.1
---> Installing application source...
---> Installing dependencies ...
Collecting scikit-learn==0.19.0 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/a4/b3/209652a5d60ce4a2a8a35ad893d7565bbb0f87ce043264ba5c9e7de304cd/scikit_learn-0.19.0-cp36-cp36m-manylinux1_x86_64.whl (12.4MB)
Collecting scipy==0.18.1 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/74/c0/f0bf4eaef1b6aa7bdd1ae5597ce1d9e729417b3ca085c47d0f1c640d34f8/scipy-0.18.1-cp36-cp36m-manylinux1_x86_64.whl (42.5MB)
Build completed successfully
dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ docker run --name "iris_predictor" -d --rm -p 5000:5000 sklearn-iris:0.1
bac6ad916bd9bea8f6d5bc2c22b6f41b79b9482ea057ea9d53282a65da473026
dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:
{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[7.197, 2.787, 3.166, 2.002]]}}
Traceback (most recent call last):
  File "/usr/local/bin/seldon-core-tester", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 297, in main
    run_predict(args)
  File "/usr/local/lib/python3.6/site-packages/seldon_core/tester.py", line 233, in run_predict
    jresp = response.json()
  File "/usr/local/lib/python3.6/site-packages/requests-2.21.0-py3.6.egg/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ pip3 show scikit-learn
Name: scikit-learn
Version: 0.19.0
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author: Andreas Mueller
Author-email: [email protected]
License: new BSD
Location: /usr/local/lib/python3.6/site-packages
Requires:
Required-by: sklearn
dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
bac6ad916bd9        sklearn-iris:0.1    "/s2i/bin/run"      56 seconds ago      Up 54 seconds       0.0.0.0:5000->5000/tcp   iris_predictor
dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/apps/sandbox/source-to-image/seldon-core/examples/models/sklearn_iris$ docker logs iris_predictor
starting microservice
2019-01-16 16:20:53,017 - seldon_core.microservice:main:261 - INFO:  Starting microservice.py:main2019-01-16 16:20:53,019 - seldon_core.microservice:main:292 - INFO:  Annotations: {}
2019-01-16 16:20:53,402 - seldon_core.microservice:main:323 - INFO:  REST microservice running on port 5000
2019-01-16 16:20:53,403 - seldon_core.microservice:main:354 - INFO:  Starting servers
 * Serving Flask app "seldon_core.model_microservice" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
2019-01-16 16:20:53,434 - werkzeug:_log:88 - INFO:   * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

@jklaise
Copy link
Contributor

jklaise commented Jan 16, 2019

Let's bypass the tester script and query the endpoint directly, could you post the output of curl -v -d 'json={"data":{"ndarray":[[1.0,2.0, 3.0, 4.0]]}}' http://0.0.0.0:5000/predict ?

@vincentclaes
Copy link
Author

vincentclaes commented Jan 18, 2019

@jklaise sorry for taking so long to reply! i have tried and had to add the noproxy argument

curl -v --noproxy "*" -d 'json={"data":{"ndarray":[[1.0,2.0, 3.0, 4.0]]}}' http://localhost:5000/predict

and i get the expected output! so that is great. but how do i circumvent a proxy with the seldon-core-tester?

@vincentclaes
Copy link
Author

@jklaise one way to do this is to set

export http_proxy=''

and then run the seldon-core-tester command

seldon-core-tester contract.json 0.0.0.0 5000 -p

and i get the output

dvcl643@10NB03610:/mnt/c/Users/dvcl643/workspace/tools/seldon-core/examples/models/sklearn_iris$ seldon-core-tester contract.json 0.0.0.0 5000 -p
----------------------------------------
SENDING NEW REQUEST:
{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[5.169, 2.388, 2.63, 0.955]]}}
RECEIVED RESPONSE:
{'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[0.38895032827210674, 0.598546327181879, 0.012503344546014513]]}, 'meta': {}}

Time 0.01633930206298828

i think this will be useful for other people working in an enterprise environment :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants