Outlier example #1306

RafalSkolasinski · 2019-12-27T13:17:45Z

New example of seldon deployment of loan classifier with outlier detection.

~~This is still WIP. Right now there are two approaches to this tutorial:~~

~~first with all code defined in notebook that uses %%writefile to populate python modules~~
second that contains most of the relevant code already in python modules and also uses combiner in seldon graph

I am keen to go with the second approach as it seems easier to navigate, reader is less overwhelmed by lengthy notebook and code is easier to version control and reuse.

What's missing:

decide which approach should be left
choose examples that better illustrates outliers detection and tells better story
if going with second layout -> add comments
if going with first layout -> add combiner
in both cases -> make new image of tutorial graph
once everything else is ready -> test with Minikube and remove reference to docker images on docker hub in favour of local images

review-notebook-app · 2019-12-27T13:17:51Z

Check out this pull request on

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

seldondev · 2019-12-27T13:18:54Z

Fri Dec 27 13:18:53 UTC 2019
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=1

seldondev · 2019-12-27T13:19:01Z

Fri Dec 27 13:19:00 UTC 2019
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=2

seldondev · 2019-12-27T14:41:53Z

Fri Dec 27 14:41:52 UTC 2019
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=3

seldondev · 2019-12-27T14:42:02Z

Fri Dec 27 14:42:01 UTC 2019
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=4

axsaucedo · 2019-12-30T00:21:09Z

There are quite a lot of areas with "Notes from Rafal" which I assume would be removed, but several have content that may be worth keeping - would this be something you're also planning to add in this PR?

RafalSkolasinski · 2019-12-30T00:33:38Z

Ou... Sorry about those. They're in the Kubeflow example - I wanted to keep those notes in separate branch of my fork and had to start working on this PR with a wrong base by accident. Please disregard this first commit for now, I will rebase on top of master once I'm back leaving only relevant changes here. BTW, what do you think about the two approaches to the actual outliers example, which one you think is better?

axsaucedo

Nice one! Looks solid, I just added a few comments, would be good to hear your thoughts

examples/outliers/alibi-detect-combiner/pipeline/combiner/Combiner.py

examples/outliers/alibi-detect-combiner/pipeline/outliersdetector/Detector.py

axsaucedo · 2019-12-30T02:45:43Z

Ou... Sorry about those. They're in the Kubeflow example - I wanted to keep those notes in separate branch of my fork and had to start working on this PR with a wrong base by accident. Please disregard this first commit for now, I will rebase on top of master once I'm back leaving only relevant changes here. BTW, what do you think about the two approaches to the actual outliers example, which one you think is better?

Got it, sounds good. I was going to add a few more comments on the /kubeflow/ dir but I'll skip it

In regards to the approaches, I can see in the notebook that the way you show it is through the Model wrapper + remote_detect_fn, but you also include the method with the combiner. I think seeing the code for the combiner, it actually seems more intuitive in regards to the separation of concerns, as you actually build the wrapper for the detector and model separately and then use the combiner to just aggregate them. Maybe it would actually be best to go for the latter one, and providing a bit of an intro to the concept of the combiner. Looks solid!

RafalSkolasinski · 2020-01-02T12:03:04Z

Maybe it would actually be best to go for the latter one, and providing a bit of an intro to the concept of the combiner. Looks solid!

Glad to hear! I will cleanup the first approach and rebase on current master.
Shall I squash commits for cleaner history or leave them so we have other approach in git too?

seldondev · 2020-01-02T13:35:19Z

Thu Jan 2 13:35:18 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=6

seldondev · 2020-01-02T13:35:25Z

Thu Jan 2 13:35:24 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=5

RafalSkolasinski · 2020-01-02T14:25:01Z

@axsaucedo Currently the tutorial consists of three s2i builds which take some time. Most of it is installing python dependencies. What do you think about adding a Dockerfile that would pip install requirements-dev.txt and each of the three s2i builds would only add necessary model files?

This would be similar to spacy tokernizer from Kubeflow example.

seldondev · 2020-01-02T14:49:38Z

Thu Jan 2 14:49:36 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=7

seldondev · 2020-01-02T14:49:49Z

Thu Jan 2 14:49:48 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=8

seldondev · 2020-01-02T15:30:59Z

Thu Jan 2 15:30:58 UTC 2020
The logs for [lint] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=10

seldondev · 2020-01-02T15:36:23Z

Thu Jan 2 15:36:22 UTC 2020
The logs for [pr-build] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=11

RafalSkolasinski · 2020-01-02T15:36:27Z

What do you think about adding a Dockerfile that would pip install requirements-dev.txt and each of the three s2i builds would only add necessary model files?

I tried it in 6a5a714 and seems to work quite nice!

seldondev · 2020-01-02T15:36:31Z

Thu Jan 2 15:36:30 UTC 2020
The logs for [lint] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=12

seldondev · 2020-01-02T15:58:48Z

Thu Jan 2 15:58:47 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=13

seldondev · 2020-01-02T15:58:53Z

Thu Jan 2 15:58:52 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=14

seldondev · 2020-01-03T14:24:09Z

Fri Jan 3 14:24:08 UTC 2020
The logs for [pr-build] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=15

seldondev · 2020-01-03T14:24:24Z

Fri Jan 3 14:24:23 UTC 2020
The logs for [lint] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=16

seldondev · 2020-01-03T14:27:56Z

Fri Jan 3 14:27:55 UTC 2020
The logs for [lint] [18] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/18.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=18

seldondev · 2020-01-03T14:28:02Z

Fri Jan 3 14:28:01 UTC 2020
The logs for [pr-build] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=17

examples/outliers/alibi-detect-combiner/notebook.ipynb

axsaucedo

Another thing that would still be required is to add the reference in the docs so it actually appears in the documentation. For this you will have to add an nblink file with a reference to the notebook, and a link to the nblink file in the examples page.

RafalSkolasinski · 2020-01-07T15:59:45Z

For this you will have to add an nblink file with a reference to the notebook, and a link to the nblink file in the examples page.

Done!

seldondev · 2020-01-07T16:00:28Z

Tue Jan 7 16:00:27 UTC 2020
The logs for [pr-build] [19] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/19.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=19

seldondev · 2020-01-07T16:00:39Z

Tue Jan 7 16:00:38 UTC 2020
The logs for [lint] [20] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1306/20.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1306 --build=20

axsaucedo · 2020-01-07T16:07:30Z

/lgtm
/approve

seldondev · 2020-01-07T16:08:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev added the do-not-merge/work-in-progress label Dec 27, 2019

RafalSkolasinski requested a review from axsaucedo December 27, 2019 13:17

seldondev requested review from gsunner and jklaise December 27, 2019 13:17

seldondev added the size/XXL label Dec 27, 2019

axsaucedo reviewed Dec 30, 2019

View reviewed changes

RafalSkolasinski added 9 commits January 2, 2020 13:24

first try

d11771c

add preprocessor and structure notebook

e73e77f

pack outlier detection into seldon deployment

fb04000

add endpoint that combines the classification and outlier detection

0243b23

polish example and return outliers score via tags

72d63c8

cleanup model wrapper

63870fb

push alternative layout of the example

d9e4d9e

add combiner to the example

b936da7

add comments in new notebook

463de0d

use jsonData instead of strData for return values

a2839d4

remove redundant version of the example

9e46661

adjust image names

46935de

RafalSkolasinski requested a review from axsaucedo January 2, 2020 16:00

add images and remove output from requirement installation cells

1b3a77c

RafalSkolasinski changed the title ~~WIP: Outlier example~~ Outlier example Jan 6, 2020

seldondev removed the do-not-merge/work-in-progress label Jan 6, 2020

axsaucedo reviewed Jan 6, 2020

View reviewed changes

examples/outliers/alibi-detect-combiner/notebook.ipynb Show resolved Hide resolved

axsaucedo suggested changes Jan 6, 2020

View reviewed changes

seldondev assigned axsaucedo Jan 6, 2020

remove note about missing graph, add nblink

c233a66

seldondev added lgtm approved labels Jan 7, 2020

axsaucedo approved these changes Jan 7, 2020

View reviewed changes

seldondev merged commit f89347d into SeldonIO:master Jan 7, 2020

axsaucedo removed their assignment May 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Outlier example #1306

Outlier example #1306

RafalSkolasinski commented Dec 27, 2019 •

edited

Loading

review-notebook-app bot commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

axsaucedo commented Dec 30, 2019

RafalSkolasinski commented Dec 30, 2019 via email

axsaucedo left a comment

axsaucedo commented Dec 30, 2019

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

axsaucedo left a comment

RafalSkolasinski commented Jan 7, 2020

seldondev commented Jan 7, 2020

seldondev commented Jan 7, 2020

axsaucedo commented Jan 7, 2020

seldondev commented Jan 7, 2020

Outlier example #1306

Outlier example #1306

Conversation

RafalSkolasinski commented Dec 27, 2019 • edited Loading

review-notebook-app bot commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

seldondev commented Dec 27, 2019

axsaucedo commented Dec 30, 2019

RafalSkolasinski commented Dec 30, 2019 via email

axsaucedo left a comment

Choose a reason for hiding this comment

axsaucedo commented Dec 30, 2019

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

RafalSkolasinski commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 2, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

seldondev commented Jan 3, 2020

axsaucedo left a comment

Choose a reason for hiding this comment

RafalSkolasinski commented Jan 7, 2020

seldondev commented Jan 7, 2020

seldondev commented Jan 7, 2020

axsaucedo commented Jan 7, 2020

seldondev commented Jan 7, 2020

RafalSkolasinski commented Dec 27, 2019 •

edited

Loading