add support for runtime metrics #2442

RafalSkolasinski · 2020-09-15T19:28:30Z

What this PR does / why we need it:

Adds support for thread/process safe runtime metrics / tags.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Introduced seldon_core.user_model.SeldonResponse class that can be used as output of predict and other methods on User Models in order to set runtime metrics and tags.
For example

from seldon_core.user_model import SeldonResponse

class Model:
    ...
    def predict(self, features, names=[], meta=[]):
        metrics = [{"type": "COUNTER", "key": "predictions_counter", "value": 1}]
        tags = {"a": 42, "z": 5}
        return SeldonResponse(data=features.tolist(), metrics=metrics, tags=tags)

Does this PR introduce a user-facing change?:

Added support for runtime metrics and tags.

seldondev · 2020-09-15T19:30:21Z

Tue Sep 15 19:30:17 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=2

seldondev · 2020-09-15T19:30:30Z

Tue Sep 15 19:30:29 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=1

seldondev · 2020-09-15T21:25:01Z

Tue Sep 15 21:24:57 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=3

seldondev · 2020-09-15T21:25:02Z

Tue Sep 15 21:25:00 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=4

RafalSkolasinski · 2020-09-15T21:26:10Z

/test integration

seldondev · 2020-09-15T21:28:18Z

Tue Sep 15 21:28:12 UTC 2020
The logs for [integration] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=5

RafalSkolasinski · 2020-09-15T23:27:16Z

/retest

seldondev · 2020-09-15T23:28:57Z

Tue Sep 15 23:28:55 UTC 2020
The logs for [integration] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=6

python/seldon_core/seldon_methods.py

axsaucedo · 2020-09-16T06:50:06Z

python/seldon_core/user_model.py

@@ -123,6 +123,18 @@ def init_metadata(self) -> Dict:
        raise SeldonNotImplementedError("init_metadata is not implemented")


+class SeldonPrediction:


Name should change to something more specific, as this is returned in aggregate, transform_, etc

How would you suggest to name it?

Maybe SeldonResponse or ClientResponse.

Went with ClientResponse, can still adjust easily

python/seldon_core/user_model.py

RafalSkolasinski · 2020-09-16T09:47:15Z

/test integration

seldondev · 2020-09-16T09:47:19Z

Wed Sep 16 09:47:15 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=7

seldondev · 2020-09-16T09:47:19Z

Wed Sep 16 09:47:16 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=8

seldondev · 2020-09-16T09:48:56Z

Wed Sep 16 09:48:54 UTC 2020
The logs for [integration] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=9

seldondev · 2020-09-16T15:52:45Z

Wed Sep 16 15:52:43 UTC 2020
The logs for [pr-build] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=10

seldondev · 2020-09-16T15:53:00Z

Wed Sep 16 15:52:59 UTC 2020
The logs for [lint] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=11

RafalSkolasinski · 2020-09-16T15:55:05Z

/test integration

seldondev · 2020-09-16T15:57:25Z

Wed Sep 16 15:57:23 UTC 2020
The logs for [integration] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=12

seldondev · 2020-09-16T18:30:12Z

Wed Sep 16 18:30:11 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=13

seldondev · 2020-09-16T18:30:18Z

Wed Sep 16 18:30:17 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=14

axsaucedo

Looks good, final things as discussed:

Update ClientResponse -> SeldonResponse
Add nblink to example and list on notebooks.rst
Add brief mention on this in the python module docs
Add point on the existing issue on updating python documentation to add further insights

seldondev · 2020-09-17T12:46:11Z

Thu Sep 17 12:46:09 UTC 2020
The logs for [pr-build] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=15

seldondev · 2020-09-17T12:46:18Z

Thu Sep 17 12:46:17 UTC 2020
The logs for [lint] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=16

seldondev · 2020-09-17T17:40:50Z

Thu Sep 17 17:40:45 UTC 2020
The logs for [pr-build] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=17

seldondev · 2020-09-17T17:41:17Z

Thu Sep 17 17:41:16 UTC 2020
The logs for [notebooks] [19] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/19.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=19

seldondev · 2020-09-17T17:43:21Z

Thu Sep 17 17:43:18 UTC 2020
The logs for [integration] [20] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/20.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=20

axsaucedo · 2020-09-17T18:07:59Z

/approve

axsaucedo · 2020-09-17T18:08:18Z

@RafalSkolasinski it seems the lint job failed, may need rebase

seldondev · 2020-09-18T09:33:37Z

New changes are detected. LGTM label has been removed.

RafalSkolasinski · 2020-09-18T09:34:06Z

/test integration

seldondev · 2020-09-18T09:34:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev · 2020-09-18T09:35:20Z

Fri Sep 18 09:35:19 UTC 2020
The logs for [pr-build] [21] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/21.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=21

seldondev · 2020-09-18T09:35:30Z

Fri Sep 18 09:35:29 UTC 2020
The logs for [lint] [22] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/22.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=22

seldondev · 2020-09-18T09:35:43Z

Fri Sep 18 09:35:42 UTC 2020
The logs for [integration] [23] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/23.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=23

RafalSkolasinski · 2020-09-18T09:36:04Z

Just so we can track failed tests.

Integration: test_label_update[1.0.2]
Note: most likely just old good flakiness

Notebooks: TestNotebooks.test_grpc_metadat and TestNotebooks.test_explainer_examples

seldondev · 2020-09-18T09:42:28Z

@RafalSkolasinski: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
notebooks	`b7b33ea`	link	`/test notebooks`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

RafalSkolasinski · 2020-09-18T09:44:27Z

/test integration

seldondev · 2020-09-18T09:45:53Z

Fri Sep 18 09:45:48 UTC 2020
The logs for [pr-build] [24] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/24.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=24

seldondev · 2020-09-18T09:45:59Z

Fri Sep 18 09:45:58 UTC 2020
The logs for [lint] [25] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/25.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=25

seldondev · 2020-09-18T09:46:07Z

Fri Sep 18 09:46:06 UTC 2020
The logs for [integration] [26] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2442/26.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2442 --build=26

seldondev added the size/XL label Sep 15, 2020

RafalSkolasinski marked this pull request as draft September 15, 2020 19:28

seldondev added the do-not-merge/work-in-progress label Sep 15, 2020

RafalSkolasinski marked this pull request as ready for review September 15, 2020 21:26

seldondev removed the do-not-merge/work-in-progress label Sep 15, 2020

axsaucedo suggested changes Sep 16, 2020

View reviewed changes

seldondev assigned axsaucedo Sep 16, 2020

seldondev added size/XXL and removed size/XL labels Sep 16, 2020

axsaucedo reviewed Sep 17, 2020

View reviewed changes

axsaucedo approved these changes Sep 17, 2020

View reviewed changes

seldondev added the lgtm label Sep 17, 2020

seldondev added the approved label Sep 17, 2020

RafalSkolasinski added 7 commits September 18, 2020 10:33

add support for runtime metrics

e401138

add support for runtime tags

a63f80f

code review suggestions

4a9c04b

code review suggestions #2

0896531

add runtime metrics/tags example

6472e24

rename CLientResponse to SeldonResponse

1487d10

code review suggestions #3

6b184b7

seldondev removed the lgtm label Sep 18, 2020

make fmt

9c9d74f

seldondev merged commit 8eb6dcb into SeldonIO:master Sep 18, 2020

		@@ -123,6 +123,18 @@ def init_metadata(self) -> Dict:
		raise SeldonNotImplementedError("init_metadata is not implemented")


		class SeldonPrediction:

add support for runtime metrics #2442

add support for runtime metrics #2442

Conversation

RafalSkolasinski commented Sep 15, 2020 • edited Loading

seldondev commented Sep 15, 2020

seldondev commented Sep 15, 2020

seldondev commented Sep 15, 2020

seldondev commented Sep 15, 2020

RafalSkolasinski commented Sep 15, 2020

seldondev commented Sep 15, 2020

RafalSkolasinski commented Sep 15, 2020

seldondev commented Sep 15, 2020

axsaucedo Sep 16, 2020

Choose a reason for hiding this comment

RafalSkolasinski Sep 16, 2020

Choose a reason for hiding this comment

RafalSkolasinski Sep 16, 2020

Choose a reason for hiding this comment

RafalSkolasinski Sep 16, 2020

Choose a reason for hiding this comment

RafalSkolasinski commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

RafalSkolasinski commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

seldondev commented Sep 16, 2020

axsaucedo left a comment

Choose a reason for hiding this comment

seldondev commented Sep 17, 2020

seldondev commented Sep 17, 2020

seldondev commented Sep 17, 2020

seldondev commented Sep 17, 2020

seldondev commented Sep 17, 2020

axsaucedo commented Sep 17, 2020

axsaucedo commented Sep 17, 2020

seldondev commented Sep 18, 2020

RafalSkolasinski commented Sep 18, 2020

seldondev commented Sep 18, 2020

seldondev commented Sep 18, 2020

seldondev commented Sep 18, 2020

seldondev commented Sep 18, 2020

RafalSkolasinski commented Sep 18, 2020

seldondev commented Sep 18, 2020 • edited Loading

RafalSkolasinski commented Sep 18, 2020

seldondev commented Sep 18, 2020

seldondev commented Sep 18, 2020

seldondev commented Sep 18, 2020

RafalSkolasinski commented Sep 15, 2020 •

edited

Loading

seldondev commented Sep 18, 2020 •

edited

Loading