Allow both http and grpc #2574

ukclivecox · 2020-10-24T18:08:25Z

What this PR does / why we need it:

Updates python wrapper, operator, executor, prepackaged servers to support both REST and gRPC at same time

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Python models will need to be rebuilt to take advantage of the exposing of both HTTP and gRPC.
Upgrading the operator will cause an upgrade to all models running.

review-notebook-app · 2020-10-24T18:08:30Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

seldondev · 2020-10-25T11:52:22Z

Sun Oct 25 11:52:17 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=3

seldondev · 2020-10-25T11:52:27Z

Sun Oct 25 11:52:27 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=4

ukclivecox · 2020-10-25T12:11:44Z

/test integration

seldondev · 2020-10-25T12:13:10Z

Sun Oct 25 12:13:09 UTC 2020
The logs for [integration] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=5

ukclivecox · 2020-10-25T15:03:33Z

/test integration

seldondev · 2020-10-25T15:05:03Z

Sun Oct 25 15:05:02 UTC 2020
The logs for [integration] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=6

ukclivecox · 2020-10-25T17:24:25Z

/test integration

seldondev · 2020-10-25T17:25:52Z

Sun Oct 25 17:25:49 UTC 2020
The logs for [integration] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=7

ukclivecox · 2020-10-25T18:21:54Z

/test integration

seldondev · 2020-10-25T18:22:40Z

Sun Oct 25 18:22:39 UTC 2020
The logs for [lint] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=9

seldondev · 2020-10-25T18:22:53Z

Sun Oct 25 18:22:53 UTC 2020
The logs for [pr-build] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=8

seldondev · 2020-10-25T18:24:46Z

Sun Oct 25 18:24:44 UTC 2020
The logs for [integration] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=10

ukclivecox · 2020-10-26T10:44:31Z

/test integration

seldondev · 2020-10-26T10:45:33Z

Mon Oct 26 10:45:32 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=13

seldondev · 2020-10-26T10:45:37Z

Mon Oct 26 10:45:36 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=14

seldondev · 2020-10-26T10:47:39Z

Mon Oct 26 10:47:38 UTC 2020
The logs for [integration] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=15

ukclivecox · 2020-10-26T13:38:15Z

/test integration

seldondev · 2020-10-26T13:39:28Z

Mon Oct 26 13:39:27 UTC 2020
The logs for [pr-build] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=16

seldondev · 2020-10-26T13:40:17Z

Mon Oct 26 13:40:15 UTC 2020
The logs for [integration] [18] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/18.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=18

RafalSkolasinski · 2020-11-11T09:34:18Z

Checking backward compatibility now: took a model that was working fine for 1.4.0 in both gRPC and REST flavour.

Built with wrapper 1.4.0 and deployed with this branch's operator / executor:

REST flavour works completely fine
gRPC does not start -> debugging now

Built with this branch's wrapper:

REST flavour does not start: seldon-core-microservice: error: unrecognized arguments: REST
gRPC flavour does not start: seldon-core-microservice: error: unrecognized arguments: GRPC

RafalSkolasinski · 2020-11-11T09:44:50Z

Built with this branch's wrapper:

REST flavour does not start: seldon-core-microservice: error: unrecognized arguments: REST

gRPC flavour does not start: seldon-core-microservice: error: unrecognized arguments: GRPC

We could allow second positional argument to be set, ignore it, and log deprecation warning.
This is probably only issue if someone is building image using Dockerfile and not s2i mechanism though.

RafalSkolasinski

Massive but extremely useful addition!

doc/source/reference/upgrading.md

RafalSkolasinski · 2020-11-11T10:32:22Z

notebooks/backwards_compatability.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Backwards Compatability Examples with Different Protocols\n",


Extending notebook to cover gRPC will be useful

RafalSkolasinski · 2020-11-11T10:41:31Z

examples/models/sklearn_iris/sklearn_iris.ipynb

@@ -19,138 +19,9 @@
    "pip install seldon-core\n",


Section about training locally and wrapping to test with Docker was quite useful, IMO. Very introductory to wrapping models.

Reply via ReviewNB

RafalSkolasinski · 2020-11-11T10:41:32Z

notebooks/backwards_compatability.ipynb

@@ -0,0 +1,985 @@
+{


Not sure what is prepackaged iris testing here, it will be just deployed using old server image so it's just
"old operator + "old sklearn server"

Reply via ReviewNB

RafalSkolasinski · 2020-11-11T10:41:32Z

notebooks/max_grpc_msg_size.ipynb

@@ -20,6 +20,20 @@
    "```"


Nice one, may be worth contributing back this magic to Jupyter :)

Reply via ReviewNB

RafalSkolasinski · 2020-11-11T10:56:55Z

python/seldon_core/microservice.py

@@ -228,7 +204,6 @@ def main():
    sys.path.append(os.getcwd())
    parser = argparse.ArgumentParser()
    parser.add_argument("interface_name", type=str, help="Name of the user interface.")
-    parser.add_argument("api_type", type=str, choices=["REST", "GRPC", "FBS"])


We should mention removal of api_type argument in upgrading section.

RafalSkolasinski · 2020-11-11T10:57:12Z

python/seldon_core/microservice.py

@@ -298,10 +273,17 @@ def main():
    )

    parser.add_argument(
-        "--port",


Same as above, we should mention change in wrapper API in upgrading section.

RafalSkolasinski · 2020-11-11T14:50:53Z

Built with wrapper 1.4.0 and deployed with this branch's operator / executor:

REST flavour works completely fine

gRPC does not start -> debugging now

It seems that in case of gRPC model it is instructed properly to serve gRPC traffic on port 9500 but executor tries to check 9000/ 9001:

{"level":"error","ts":1605106209.0915442,"logger":"SeldonRestApi","msg":"Ready check failed","error":"dial tcp [::1]:9001: connect: connection refused" .....

(it's 9001 as I am looking at two-node model)

seldondev · 2020-11-12T08:20:55Z

Thu Nov 12 08:20:53 UTC 2020
The logs for [pr-build] [72] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/72.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=72

seldondev · 2020-11-12T08:21:00Z

Thu Nov 12 08:20:59 UTC 2020
The logs for [lint] [73] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/73.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=73

seldondev · 2020-11-12T14:42:42Z

Thu Nov 12 14:42:38 UTC 2020
The logs for [pr-build] [74] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/74.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=74

seldondev · 2020-11-12T14:42:50Z

Thu Nov 12 14:42:48 UTC 2020
The logs for [lint] [75] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/75.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=75

seldondev · 2020-11-12T14:43:46Z

Thu Nov 12 14:43:45 UTC 2020
The logs for [pr-build] [76] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/76.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=76

seldondev · 2020-11-12T14:44:02Z

Thu Nov 12 14:44:01 UTC 2020
The logs for [lint] [77] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/77.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=77

ukclivecox · 2020-11-12T14:57:09Z

/test integration

seldondev · 2020-11-12T14:58:53Z

Thu Nov 12 14:58:52 UTC 2020
The logs for [integration] [78] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/78.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=78

ukclivecox · 2020-11-13T07:50:41Z

/test notebooks

seldondev · 2020-11-13T07:52:07Z

Fri Nov 13 07:52:06 UTC 2020
The logs for [notebooks] [79] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/79.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=79

axsaucedo · 2020-11-16T14:14:17Z

/approve

seldondev · 2020-11-16T14:14:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev · 2020-11-16T14:16:59Z

Mon Nov 16 14:16:57 UTC 2020
The logs for [lint] [81] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/81.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=81

seldondev · 2020-11-16T14:17:07Z

Mon Nov 16 14:17:04 UTC 2020
The logs for [pr-build] [80] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2574/80.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2574 --build=80

ukclivecox added 2 commits October 21, 2020 08:25

Initial work to allow both REST and gRPC

7fbfd94

updates and test

9d256e5

seldondev added the do-not-merge/work-in-progress label Oct 24, 2020

seldondev added the size/XXL label Oct 24, 2020

simon-cj mentioned this pull request Oct 25, 2020

[upstream] Support GPRC and HTTP at the same time kleveross/klever-model-registry#86

Closed

rebase and fix microservice tests

166379c

update to 1.4.0-dev as some references were still 1.3.0-dev

0bf6289

ukclivecox added 2 commits October 26, 2020 10:39

Fix istio tests and helm tests

3c08670

rebase

0b017bc

Fix istio patch

3d09623

RafalSkolasinski suggested changes Nov 11, 2020

View reviewed changes

seldondev assigned RafalSkolasinski Nov 11, 2020

RafalSkolasinski reviewed Nov 11, 2020

View reviewed changes

RafalSkolasinski suggested changes Nov 11, 2020

View reviewed changes

add backwards compatability for old grpc servers

a74b493

ukclivecox added 2 commits November 12, 2020 14:41

Updates from reviews

e068dfc

fmt

6ef2187

axsaucedo requested review from axsaucedo, adriangonz and RafalSkolasinski November 16, 2020 14:13

seldondev added the approved label Nov 16, 2020

adriangonz merged commit b5e64ad into SeldonIO:master Nov 16, 2020

ukclivecox mentioned this pull request Nov 17, 2020

Add sideEffects to Webhooks to allow Dry Runs #2651

Merged

Allow both http and grpc #2574

Allow both http and grpc #2574

Conversation

ukclivecox commented Oct 24, 2020 • edited Loading

review-notebook-app bot commented Oct 24, 2020

seldondev commented Oct 25, 2020

seldondev commented Oct 25, 2020

ukclivecox commented Oct 25, 2020

seldondev commented Oct 25, 2020

ukclivecox commented Oct 25, 2020

seldondev commented Oct 25, 2020

ukclivecox commented Oct 25, 2020

seldondev commented Oct 25, 2020

ukclivecox commented Oct 25, 2020

seldondev commented Oct 25, 2020

seldondev commented Oct 25, 2020

seldondev commented Oct 25, 2020

ukclivecox commented Oct 26, 2020

seldondev commented Oct 26, 2020

seldondev commented Oct 26, 2020

seldondev commented Oct 26, 2020

ukclivecox commented Oct 26, 2020

seldondev commented Oct 26, 2020

seldondev commented Oct 26, 2020

RafalSkolasinski commented Nov 11, 2020 • edited Loading

RafalSkolasinski commented Nov 11, 2020

RafalSkolasinski left a comment

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

ukclivecox Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 11, 2020

Choose a reason for hiding this comment

RafalSkolasinski commented Nov 11, 2020 • edited Loading

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

ukclivecox commented Nov 12, 2020

seldondev commented Nov 12, 2020

ukclivecox commented Nov 13, 2020

seldondev commented Nov 13, 2020

axsaucedo commented Nov 16, 2020

seldondev commented Nov 16, 2020

seldondev commented Nov 16, 2020

seldondev commented Nov 16, 2020

ukclivecox commented Oct 24, 2020 •

edited

Loading

RafalSkolasinski commented Nov 11, 2020 •

edited

Loading

RafalSkolasinski commented Nov 11, 2020 •

edited

Loading