sklearn iris model incompatible with latest sklearnserver (1.10.0-dev)

[RafalSkolasinski]

Describe the bug

It seems that our model

gs://seldon-models/sklearn/iris

is now incompatible with new version of seldonio/sklearnserver:1.10.0-dev.

This is due to scikit-learn update 0.23.2 -> 0.24.2

To reproduce

1. Install 1.10.0-dev Seldon Core
2. Deploy

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn
spec:
  predictors:
    - graph:
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/sklearn/iris
        name: classifier
      name: default
      replicas: 1

or deploy explicitly this version of server on any (recent) version of Seldon Core

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn
spec:
  predictors:
    - graph:
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/sklearn/iris
        name: classifier
      name: default
      replicas: 1
        
      componentSpecs:
      - spec:
          containers:
          - name: classifier
            image: seldonio/sklearnserver:1.10.0-dev

3, observe error

starting microservice
2021-07-23 10:26:09,177 - seldon_core.microservice:main:206 - INFO:  Starting microservice.py:main
2021-07-23 10:26:09,177 - seldon_core.microservice:main:207 - INFO:  Seldon Core version: 1.10.0-dev
2021-07-23 10:26:09,178 - seldon_core.microservice:main:362 - INFO:  Parse JAEGER_EXTRA_TAGS []
2021-07-23 10:26:09,178 - seldon_core.microservice:load_annotations:158 - INFO:  Found annotation kubernetes.io/config.seen:2021-07-23T10:24:29.964141761Z 
2021-07-23 10:26:09,178 - seldon_core.microservice:load_annotations:158 - INFO:  Found annotation kubernetes.io/config.source:api 
2021-07-23 10:26:09,178 - seldon_core.microservice:load_annotations:158 - INFO:  Found annotation prometheus.io/path:/prometheus 
2021-07-23 10:26:09,178 - seldon_core.microservice:load_annotations:158 - INFO:  Found annotation prometheus.io/scrape:true 
2021-07-23 10:26:09,178 - seldon_core.microservice:main:365 - INFO:  Annotations: {'kubernetes.io/config.seen': '2021-07-23T10:24:29.964141761Z', 'kubernetes.io/config.source': 'api', 'prometheus.io/path': '/prometheus', 'prometheus.io/scrape': 'true'}
2021-07-23 10:26:09,178 - seldon_core.microservice:main:369 - INFO:  Importing SKLearnServer
2021-07-23 10:26:09,193 - SKLearnServer:__init__:21 - INFO:  Model uri: /mnt/models
2021-07-23 10:26:09,193 - SKLearnServer:__init__:22 - INFO:  method: predict_proba
2021-07-23 10:26:09,193 - SKLearnServer:load:26 - INFO:  load
2021-07-23 10:26:09,193 - root:download:31 - INFO:  Copying contents of /mnt/models to local
2021-07-23 10:26:09,193 - SKLearnServer:load:30 - INFO:  model file: /mnt/models/model.joblib
Traceback (most recent call last):
  File "/opt/conda/bin/seldon-core-microservice", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.7/site-packages/seldon_core/microservice.py", line 379, in main
    user_object = user_class(**parameters)
  File "/microservice/SKLearnServer.py", line 23, in __init__
    self.load()
  File "/microservice/SKLearnServer.py", line 31, in load
    self._joblib = joblib.load(model_file)
  File "/opt/conda/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 585, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/opt/conda/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 504, in _unpickle
    obj = unpickler.load()
  File "/opt/conda/lib/python3.7/pickle.py", line 1088, in load
    dispatch[key[0]](self)
  File "/opt/conda/lib/python3.7/pickle.py", line 1376, in load_global
    klass = self.find_class(module, name)
  File "/opt/conda/lib/python3.7/pickle.py", line 1426, in find_class
    __import__(module, level=0)
ModuleNotFoundError: No module named 'sklearn.linear_model.logistic'

Expected behaviour

It works.

Environment

Kind.

[RafalSkolasinski]

Most likely these integration tests are affected by this bug:

    TestBatchWorker.test_batch_worker
    TestPrepack.test_sklearn
    TestPrepack.test_text_alibi_explainer
    TestPrepack.test_openapi_sklearn

[RafalSkolasinski]

This blocks #3420 for example.

[RafalSkolasinski]

If required, a quick workaround for a problem is to explicitly specify latest stable image.

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn
spec:
  predictors:
    - graph:
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/sklearn/iris
        name: classifier
      name: default
      replicas: 1
        
      componentSpecs:
      - spec:
          containers:
          - name: classifier
            image: seldonio/sklearnserver:1.9.1

In any case, this should not affect the latest stable release at all.