seldonio/seldon-core-s2i-python3-tf-gpu:0.15 image default python version is 2.7.17

[ford-mach1ml]

Built an image with seldonio/seldon-core-s2i-python3-tf-gpu:0.15 as the base image, with requirements.txt that includes xgboost==0.82 (got an error trying to use xgboost==0.90, which may have been related to this issue.)

Upon deploying a model with the created image, got this error: ModuleNotFoundError: No module named 'xgboost'

Doing kubectl exec into the pod, it appears that xgboost is getting installed for python2, not python3:

root@confident-edge-xgboost-deployment-default-dd7d9aa-6f64776dvwvlq:/microservice# python --version
Python 2.7.17
root@confident-edge-xgboost-deployment-default-dd7d9aa-6f64776dvwvlq:/microservice# python3 --version
Python 3.6.9
root@confident-edge-xgboost-deployment-default-dd7d9aa-6f64776dvwvlq:/microservice# python3
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xgboost
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xgboost'
>>> exit()
root@confident-edge-xgboost-deployment-default-dd7d9aa-6f64776dvwvlq:/microservice# python 
Python 2.7.17 (default, Nov  7 2019, 10:07:09) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xgboost
>>> xgboost.__version__
'0.82'
>>> exit()

[RafalSkolasinski]

Thanks for submitting the issue. I can confirm that there is indeed a problem with the seldon-core-s2i-python3-tf-gpu.

I was able to pin it to the fact that our regular python images uses as their base e.g. python:3.7 whereas the gpu image uses nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04.

In python images pip always points to the right executable and on Ubuntu 18.04 unfortunately pip and pip3 refers to python 2 and python 3 respectively.

While we work on the solution for this as the workaround you could build your own base image (base it on our GPU image) and pip3 install whatever you need. This way you will make sure that it gets installed into the Python 3 site-packages. You can look at the outlier example where we do something similar (see Dockerfile and Makefile).

[RafalSkolasinski]

@axsaucedo As we build this image as python3 why do we even install Python 2 there?

seldon-core/wrappers/s2i/python/Dockerfile.gpu.tmpl

Line 7 in c687012

RUN apt-get install -y python-pip python-dev build-essential

As a fix to this we could not install Python 2 and just symlink pip and python to pip3 and python3 - making it exclusively Python 3 image. As neither of those is present in the base image this shouldn't break anything critical.

Other solution would be to setup some environmental variables and modify assemble script accordingly to use pip3 instead of pip.

[RafalSkolasinski]

I open a PR that solves it using symlinks.

[RafalSkolasinski]

@ford-mach1ml The fix for this issue has been merged to master.
Updated image has been uploaded to docker hub as seldonio/seldon-core-s2i-python3-tf-gpu:0.16-SNAPSHOT and will be part of 0.16 release.

Please, give it a shot and reopen the issue in case it does not solve your problem.

[ford-mach1ml]

Looks good to me. Thanks for taking care of this!