Custom Metrics

docs/custom_metrics.md

@@ -0,0 +1,65 @@
+# Custom Metrics
+
+Seldon Core exposes basic metrics via Prometheus endpoints on its service orchestrator that include request count, request time percentiles and rolling accuracy for each running model. However, you may wish to expose custom metrics from your components which are automaticlaly added to Prometheus. For this purpose you can supply extra fields in the returned meta data of the response object in the API calls to your components as illustrated below:
+
+```
+{
+	"meta": {
+		"metrics": [
+			{
+				"type": "COUNTER",
+				"key": "mycounter",
+				"value": 1.0
+			},
+			{
+				"type": "GAUGE",
+				"key": "mygauge",
+				"value": 22.0
+			},
+			{
+				"type": "TIMER",
+				"key": "mytimer",
+				"value": 1.0
+			}
+		]
+	},
+	"data": {
+		"ndarray": [
+			[
+				1,
+				2
+			]
+		]
+	}
+}
+```
+
+We provide three types of metric that can be returned in the meta.metrics list:
+
+ * COUNTER : a monotonically increasing value. It will be added to any existing value from the metric key.
+ * GAUGE : an absolute value showing a level, it will overwrite any existing value.
+ * TIMER : a time value (in msecs)
+
+Each metric apart from the type takes a key and a value. The proto buffer definition is shown below:
+
+```
+message Metric {
+ enum MetricType {
+     COUNTER = 0;
+     GAUGE = 1;
+     TIMER = 2;
+ }
+ string key = 1;
+ MetricType type = 2;
+ float value = 3;
+}
+```
+
+At present the following Seldon Core wrappers provide integrations with custom metrics:
+
+ * [Python Wrapper](./wrappers/python.md#custom-metrics)
+
+
+# example
+
+There is an [example notebook illustrating a model with custom metrics in python](../examples/models/template_model_with_metrics/modelWithMetrics.ipynb).

docs/production.md

@@ -3,7 +3,8 @@
 
 This page will discuss the various added functionality you might need for running Seldon Core in a production environment.
 
-
  * [Pulling from Private Docker Registries](private_registries.md)
  * [gRPC max message size](grpc_max_message_size.md)
+ * [Custom Metrics](custom_metrics.md)
+
 

docs/proposals/custom_metrics.md

@@ -12,8 +12,8 @@ Extend the SeldonMessage proto buffer to have a "metrics" element in the meta da
 {
 "meta" : {
   "metrics" : [
-    { "key" : "my_metric_1", "type" : "counter", "value" : 1 },
-    { "key" : "my_metric_2", "type" : "guage", "value" : 223 }
+    { "key" : "my_metric_1", "type" : "COUNTER", "value" : 1 },
+    { "key" : "my_metric_2", "type" : "GAUGE", "value" : 223 }
   ]
 }
 }
@@ -45,7 +45,6 @@ message Metric {
  string key = 1;
  MetricType type = 2;
  float value = 3;
- string graphId = 4;
 }
 ```
 
@@ -59,7 +58,7 @@ We use [Micrometer](https://micrometer.io) for exposing metrics. Counter and gau
 ## Engine Implementation
 
  1. For each component if there is a metrics section parse and expose via prometheus each metric of the appropriate type.
- 2. Merge all metrics into final set for returning externally adding graph id of the component that returned the metrics if missing.
+ 2. Merge all metrics into final set for returning externally 
 
 ## Wrapper Implementations
 
@@ -70,8 +69,8 @@ Add optional new function in class user defines
 ```
 def metrics(self):
   return [
-    { "key" : "my_metric_1", "type" : "counter", "value" : self.counter1 },
-    { "key" : "my_metric_2", "type" : "guage", "value" : self.guage1 }
+    { "key" : "my_metric_1", "type" : "COUNTER", "value" : self.counter1 },
+    { "key" : "my_metric_2", "type" : "GAUGE", "value" : self.guage1 }
   ]
 ```
 

docs/wrappers/python.md

@@ -171,6 +171,43 @@ Set either to 0 or 1. Default is 0. If set to 1 then your model will be saved pe
  * [Example transformers](https://github.com/SeldonIO/seldon-core/tree/master/examples/transformers)
 
 
+# Advanced Usage
+
+## Custom Metrics
+```from version 0.3```
+
+To add custom metrics to your response you can define an optional method ```metrics``` in your class that returns a list of metric dicts. An example is shown below:
+
+```
+class MyModel(object):
+
+    def predict(self,X,features_names):
+        return X
+
+    def metrics(self):
+    	return [{"type":"COUNTER","key":"mycounter","value":1}]
+```
+
+For more details on custom metrics and the format of the metric dict see [here](../custom_metrics.md).
+
+There is an [example notebook illustrating a model with custom metrics in python](../../examples/models/template_model_with_metrics/modelWithMetrics.ipynb).
+
+## Custom Meta Data
+```from version 0.3```
+
+To add custom meta data you can add an optional method ```tags``` which can return a dict of custom meta tags as shown in the example below:
+
+```
+class UserObject(object):
+
+    def predict(self,X,features_names):
+        return X
+
+    def tags(self):
+        return {"mytag":1}
+```
+
+
 
 
 

engine/src/main/java/io/seldon/engine/metrics/CustomMetricsManager.java

@@ -0,0 +1,38 @@
+package io.seldon.engine.metrics;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.springframework.stereotype.Component;
+
+import com.google.common.util.concurrent.AtomicDouble;
+
+import io.micrometer.core.instrument.Metrics;
+import io.seldon.engine.predictors.PredictiveUnitState;
+import io.seldon.protos.PredictionProtos.Metric;
+
+/**
+ * 
+ * @author clive
+ * Handles the storage of gauges for custom metrics.
+ *
+ */
+@Component
+public class CustomMetricsManager {
+
+	private ConcurrentHashMap<PredictiveUnitState,AtomicDouble> gauges = new ConcurrentHashMap<>();
+
+	public AtomicDouble get(PredictiveUnitState state,Metric metric)
+	{
+		if (gauges.containsKey(state))
+			return gauges.get(state);
+		else
+		{
+			AtomicDouble d = new AtomicDouble();
+			gauges.put(state, d);
+			Metrics.globalRegistry.gauge(metric.getKey(), d);
+			return d;
+		}
+	}
+
+
+}

engine/src/main/java/io/seldon/engine/predictors/EnginePredictor.java

@@ -118,6 +118,15 @@ public String getDeploymentName() {
 		return deploymentName;
 	}
 
+
+    /**
+     * Used only for testing. Should be replaced by better methods that use Spring and Mockito to create a PredictorSpec for testing
+     * @param predictorSpec
+     */
+	public void setPredictorSpec(PredictorSpec predictorSpec) { //FIXME
+		this.predictorSpec = predictorSpec;
+	}
+
 	private static PredictorSpec buildDefaultPredictorSpec() {
 
         //@formatter:off