Define future approach to metadata and metrics with new Executor

[RafalSkolasinski]

What this issue is about

Very soon Executor, in Go replacement for currently used Java engine, will be merged into master.

In this context we need to define our future approach to metadata and metrics. How it is gonna be ideally handled in the future and how are we gonna avoid breaking backward compatibility during period of time when we support both Executor and Engine.

In order to help discussion I did try to put together a summary of the current state of matter.
As the topic seems to cause some confusion, especially on the semantic level, the description of this issue may be a bit lengthy, apologies for that.

Definitions

Metadata (model):

• Model specific information: model name, author, training dataset, input's shape and type, etc...
• May or may not be dynamic (depending on whether it is exposed)
• Does not depend on incoming requests

Metadata (request):

• This is the metadata that is sent with each request / response
• This is the metadata of the request itself (e.g. sender or unique identifier)
• May depend on external resources (e.g. external api version on time of request processing)
• In the future may be included in the header of the HTTP request
• It is part of the SeldonMessage

Metrics:

• Should represent current state of model / deployment
• Standard devops metrics (like number of requests per second, cpu usage) we provide by default and expose via endpoint for Prometheus
• User defined custom metrics is returned with requests (Seldon Engine)
• In the executor each container will expose its own metrics endpoint (picked up by Prometheus)

Now: only Java Engine

User-provided models can define two methods: metrics and tags.
Both of these are included in response.meta when /predict end point is called.
All of that logic is defined in the Python wrapper, not in Java engine.

Metrics method:

• Return type: list of dictionaries
• Directly used in user_model.py::client_custom_metrics method
• Returned in response.meta.metrics of SeldonMessage
• Commonly used to for custom metrics defined by user

Tags method:

• Return type: dictionary
• Directly used in user_model.py::client_custom_tags method
• Returned in response.meta.tags of SeldonMessage
• Commonly used to return request metadata
• Problem: predict method must modify current state of model in order to provide data returned by this method

Future: compatibility with both Executor and Engine

For some time we will be supporting both Executor and Engine.
Python wrapper must be therefore compatible with both.
Wrapper will know if it is working with Executor or Engine through environmental variables.

Go Executor:

Metadata:

• User-provided models can define a metadata method.
• Output from that method will be exposed via /metadata endpoint.
• It contains pure model metadata in the context of the above definitions.

Metrics:

• Executor’s /metadata endpoint will only contain general metrics
• Custom metrics will be exposed on /metadata of each container (and consumed by Prometheus)

Java Engine:

Main compatibility with the current functionality until we deprecate Java Engine.

Problems / objectives

• What to do with current metrics and tags method? Shall these get deprecated?”
• How to provide backward compatibility with models defined to work with pre-executor seldon-core? These may use metrics and tags in a non-trivial manner.
• Shall per-request metadata be appended to response?

Main questions:

• How do we want to handle these things (especially custom metrics) in the future?
• How do we keep compatibility with earlier defined models?

---

P.S. I did my best to get these points right. However, if something is wrong or confusing, please, leave a comment or let me know and we'll try to clarify it.

[RafalSkolasinski]

Ideally, this issue will lead us to a clear new picture on how we want to handle these things or set of well defined issues that precisely describe desired functionality.

[RafalSkolasinski]

I believe equivalent of ours request metadata in kfserving dataplane proposal would be $parameters present in $inference_request, $request_input, $request_output and $inference_response.

Actually, I think I was not exactly correct. Seems that $parameters are more about controlling inference process.

Not sure then if request metadata is in any way present in this dataplane proposal.

[RafalSkolasinski]

The request metadata is in principle metadata about the request itself.

It can be immutable data that comes as a part of the request. In example:

• some id of user who send the request (for audit purposes)
• some unique request identifier (could be useful in "Batch" processing)

This data should be exposed to predict method but guaranteed to be immutable.

The other example of request metadata provided by @adriangonz is recording version of external libraries accessed during the time of processing the request. This type of data is:

• unknown to sender on time of the request
• may change in between requests

We should provide an easy and threadsafe way to include that kind of metadata in the request response.

[RafalSkolasinski]

Reference: kfserving dataplane proposal

[ukclivecox]

already in progress in other issues