diff --git a/docs/custom_metrics.md b/docs/custom_metrics.md new file mode 100644 index 0000000000..11d1fe1065 --- /dev/null +++ b/docs/custom_metrics.md @@ -0,0 +1,65 @@ +# Custom Metrics + +Seldon Core exposes basic metrics via Prometheus endpoints on its service orchestrator that include request count, request time percentiles and rolling accuracy for each running model. However, you may wish to expose custom metrics from your components which are automaticlaly added to Prometheus. For this purpose you can supply extra fields in the returned meta data of the response object in the API calls to your components as illustrated below: + +``` +{ + "meta": { + "metrics": [ + { + "type": "COUNTER", + "key": "mycounter", + "value": 1.0 + }, + { + "type": "GAUGE", + "key": "mygauge", + "value": 22.0 + }, + { + "type": "TIMER", + "key": "mytimer", + "value": 1.0 + } + ] + }, + "data": { + "ndarray": [ + [ + 1, + 2 + ] + ] + } +} +``` + +We provide three types of metric that can be returned in the meta.metrics list: + + * COUNTER : a monotonically increasing value. It will be added to any existing value from the metric key. + * GAUGE : an absolute value showing a level, it will overwrite any existing value. + * TIMER : a time value (in msecs) + +Each metric apart from the type takes a key and a value. The proto buffer definition is shown below: + +``` +message Metric { + enum MetricType { + COUNTER = 0; + GAUGE = 1; + TIMER = 2; + } + string key = 1; + MetricType type = 2; + float value = 3; +} +``` + +At present the following Seldon Core wrappers provide integrations with custom metrics: + + * [Python Wrapper](./wrappers/python.md#custom-metrics) + + +# example + +There is an [example notebook illustrating a model with custom metrics in python](../examples/models/template_model_with_metrics/modelWithMetrics.ipynb). \ No newline at end of file diff --git a/docs/production.md b/docs/production.md index 8aa65c726b..282f4064ef 100644 --- a/docs/production.md +++ b/docs/production.md @@ -3,7 +3,8 @@ This page will discuss the various added functionality you might need for running Seldon Core in a production environment. - * [Pulling from Private Docker Registries](private_registries.md) * [gRPC max message size](grpc_max_message_size.md) + * [Custom Metrics](custom_metrics.md) + diff --git a/docs/proposals/custom_metrics.md b/docs/proposals/custom_metrics.md index 5c14f9a188..5052bd54d0 100644 --- a/docs/proposals/custom_metrics.md +++ b/docs/proposals/custom_metrics.md @@ -12,8 +12,8 @@ Extend the SeldonMessage proto buffer to have a "metrics" element in the meta da { "meta" : { "metrics" : [ - { "key" : "my_metric_1", "type" : "counter", "value" : 1 }, - { "key" : "my_metric_2", "type" : "guage", "value" : 223 } + { "key" : "my_metric_1", "type" : "COUNTER", "value" : 1 }, + { "key" : "my_metric_2", "type" : "GAUGE", "value" : 223 } ] } } @@ -45,7 +45,6 @@ message Metric { string key = 1; MetricType type = 2; float value = 3; - string graphId = 4; } ``` @@ -59,7 +58,7 @@ We use [Micrometer](https://micrometer.io) for exposing metrics. Counter and gau ## Engine Implementation 1. For each component if there is a metrics section parse and expose via prometheus each metric of the appropriate type. - 2. Merge all metrics into final set for returning externally adding graph id of the component that returned the metrics if missing. + 2. Merge all metrics into final set for returning externally ## Wrapper Implementations @@ -70,8 +69,8 @@ Add optional new function in class user defines ``` def metrics(self): return [ - { "key" : "my_metric_1", "type" : "counter", "value" : self.counter1 }, - { "key" : "my_metric_2", "type" : "guage", "value" : self.guage1 } + { "key" : "my_metric_1", "type" : "COUNTER", "value" : self.counter1 }, + { "key" : "my_metric_2", "type" : "GAUGE", "value" : self.guage1 } ] ``` diff --git a/docs/wrappers/python.md b/docs/wrappers/python.md index 576a99f7aa..c6fd6adf9a 100644 --- a/docs/wrappers/python.md +++ b/docs/wrappers/python.md @@ -171,6 +171,43 @@ Set either to 0 or 1. Default is 0. If set to 1 then your model will be saved pe * [Example transformers](https://github.com/SeldonIO/seldon-core/tree/master/examples/transformers) +# Advanced Usage + +## Custom Metrics +```from version 0.3``` + +To add custom metrics to your response you can define an optional method ```metrics``` in your class that returns a list of metric dicts. An example is shown below: + +``` +class MyModel(object): + + def predict(self,X,features_names): + return X + + def metrics(self): + return [{"type":"COUNTER","key":"mycounter","value":1}] +``` + +For more details on custom metrics and the format of the metric dict see [here](../custom_metrics.md). + +There is an [example notebook illustrating a model with custom metrics in python](../../examples/models/template_model_with_metrics/modelWithMetrics.ipynb). + +## Custom Meta Data +```from version 0.3``` + +To add custom meta data you can add an optional method ```tags``` which can return a dict of custom meta tags as shown in the example below: + +``` +class UserObject(object): + + def predict(self,X,features_names): + return X + + def tags(self): + return {"mytag":1} +``` + + diff --git a/engine/src/main/java/io/seldon/engine/metrics/CustomMetricsManager.java b/engine/src/main/java/io/seldon/engine/metrics/CustomMetricsManager.java new file mode 100644 index 0000000000..edc002d2ce --- /dev/null +++ b/engine/src/main/java/io/seldon/engine/metrics/CustomMetricsManager.java @@ -0,0 +1,38 @@ +package io.seldon.engine.metrics; + +import java.util.concurrent.ConcurrentHashMap; + +import org.springframework.stereotype.Component; + +import com.google.common.util.concurrent.AtomicDouble; + +import io.micrometer.core.instrument.Metrics; +import io.seldon.engine.predictors.PredictiveUnitState; +import io.seldon.protos.PredictionProtos.Metric; + +/** + * + * @author clive + * Handles the storage of gauges for custom metrics. + * + */ +@Component +public class CustomMetricsManager { + + private ConcurrentHashMap gauges = new ConcurrentHashMap<>(); + + public AtomicDouble get(PredictiveUnitState state,Metric metric) + { + if (gauges.containsKey(state)) + return gauges.get(state); + else + { + AtomicDouble d = new AtomicDouble(); + gauges.put(state, d); + Metrics.globalRegistry.gauge(metric.getKey(), d); + return d; + } + } + + +} diff --git a/engine/src/main/java/io/seldon/engine/predictors/EnginePredictor.java b/engine/src/main/java/io/seldon/engine/predictors/EnginePredictor.java index 4a625ba491..aee70381fd 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/EnginePredictor.java +++ b/engine/src/main/java/io/seldon/engine/predictors/EnginePredictor.java @@ -118,6 +118,15 @@ public String getDeploymentName() { return deploymentName; } + + /** + * Used only for testing. Should be replaced by better methods that use Spring and Mockito to create a PredictorSpec for testing + * @param predictorSpec + */ + public void setPredictorSpec(PredictorSpec predictorSpec) { //FIXME + this.predictorSpec = predictorSpec; + } + private static PredictorSpec buildDefaultPredictorSpec() { //@formatter:off diff --git a/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitBean.java b/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitBean.java index edae28adb7..b12396310e 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitBean.java +++ b/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitBean.java @@ -21,8 +21,11 @@ import java.util.Map; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; import org.ojalgo.matrix.PrimitiveMatrix; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.scheduling.annotation.Async; import org.springframework.scheduling.annotation.AsyncResult; @@ -32,17 +35,24 @@ import io.micrometer.core.instrument.Counter; import io.micrometer.core.instrument.Metrics; +import io.micrometer.core.instrument.Timer; import io.seldon.engine.exception.APIException; +import io.seldon.engine.metrics.CustomMetricsManager; import io.seldon.engine.metrics.SeldonRestTemplateExchangeTagsProvider; import io.seldon.engine.service.InternalPredictionService; import io.seldon.protos.DeploymentProtos.PredictiveUnit.PredictiveUnitMethod; +import io.seldon.protos.PredictionProtos.DefaultData; import io.seldon.protos.PredictionProtos.Feedback; import io.seldon.protos.PredictionProtos.Meta; +import io.seldon.protos.PredictionProtos.Metric; import io.seldon.protos.PredictionProtos.SeldonMessage; +import io.seldon.protos.PredictionProtos.Tensor; @Component public class PredictiveUnitBean extends PredictiveUnitImpl { + private final static Logger logger = LoggerFactory.getLogger(PredictiveUnitBean.class); + @Autowired InternalPredictionService internalPredictionService; @@ -52,22 +62,36 @@ public class PredictiveUnitBean extends PredictiveUnitImpl { @Autowired public PredictorConfigBean predictorConfig; + @Autowired + private CustomMetricsManager customMetricsManager; + public PredictiveUnitBean(){} public SeldonMessage getOutput(SeldonMessage request, PredictiveUnitState state) throws InterruptedException, ExecutionException, InvalidProtocolBufferException{ Map routingDict = new HashMap(); Map requestPathDict = new HashMap(); - SeldonMessage response = getOutputAsync(request, state, routingDict,requestPathDict).get(); + List metrics = new ArrayList<>(); + SeldonMessage response = getOutputAsync(request, state, routingDict,requestPathDict,metrics).get(); SeldonMessage.Builder builder = SeldonMessage .newBuilder(response) .setMeta(Meta - .newBuilder(response.getMeta()).putAllRouting(routingDict).putAllRequestPath(requestPathDict)); + .newBuilder(response.getMeta()).putAllRouting(routingDict).putAllRequestPath(requestPathDict).addAllMetrics(metrics)); return builder.build(); } + private void addMetrics(SeldonMessage msg,PredictiveUnitState state,List metrics) + { + if (msg.hasMeta()) + { + addCustomMetrics(msg.getMeta().getMetricsList(),state); + metrics.addAll(msg.getMeta().getMetricsList()); + } + } + + @Async - private Future getOutputAsync(SeldonMessage input, PredictiveUnitState state, Map routingDict,Map requestPathDict) throws InterruptedException, ExecutionException, InvalidProtocolBufferException{ + private Future getOutputAsync(SeldonMessage input, PredictiveUnitState state, Map routingDict,Map requestPathDict,List metrics) throws InterruptedException, ExecutionException, InvalidProtocolBufferException{ // This element to the request path requestPathDict.put(state.name, state.image); @@ -79,8 +103,10 @@ private Future getOutputAsync(SeldonMessage input, PredictiveUnit // Compute the transformed Input SeldonMessage transformedInput = implementation.transformInput(input, state); - // Preserve the original metadata + addMetrics(transformedInput,state,metrics); + // Preserve the original metadata except metrics transformedInput = mergeMeta(transformedInput,input.getMeta()); + if (state.children.isEmpty()){ // If this unit has no children, the transformed input becomes the output @@ -92,11 +118,19 @@ private Future getOutputAsync(SeldonMessage input, PredictiveUnit List childrenOutputs = new ArrayList(); // Get the routing. -1 means all children - int routing = implementation.route(transformedInput, state); - sanityCheckRouting(routing, state); + SeldonMessage routingMessage = implementation.route(transformedInput, state); + int routing; + if (routingMessage != null) { + routing = getBranchIndex(routingMessage, state); + sanityCheckRouting(routing, state); + addMetrics(routingMessage,state,metrics); + } else { + routing = -1; + } // Update the routing dictionary routingDict.put(state.name, routing); + if (routing == -1){ // No routing, propagate to all children selectedChildren = state.children; @@ -109,19 +143,24 @@ private Future getOutputAsync(SeldonMessage input, PredictiveUnit // Get all the children outputs asynchronously for (PredictiveUnitState childState : selectedChildren){ - deferredChildrenOutputs.add(getOutputAsync(transformedInput,childState,routingDict,requestPathDict)); + deferredChildrenOutputs.add(getOutputAsync(transformedInput,childState,routingDict,requestPathDict,metrics)); } for (Future deferredOutput : deferredChildrenOutputs){ - childrenOutputs.add(deferredOutput.get()); + SeldonMessage m = deferredOutput.get(); + childrenOutputs.add(m); } // Compute the backward transformation of all children outputs SeldonMessage aggregatedOutput = implementation.aggregate(childrenOutputs, state); + addMetrics(aggregatedOutput,state,metrics); + // Merge all the outputs metadata aggregatedOutput = mergeMeta(aggregatedOutput,childrenOutputs); SeldonMessage transformedOutput = implementation.transformOutput(aggregatedOutput, state); - // Preserve metadata + addMetrics(transformedOutput,state,metrics); + // Preserve metadata except metrics transformedOutput = mergeMeta(transformedOutput,aggregatedOutput.getMeta()); + return new AsyncResult<>(transformedOutput); @@ -205,14 +244,15 @@ public SeldonMessage aggregate(List outputs, PredictiveUnitState return outputs.get(0); } - public int route(SeldonMessage input, PredictiveUnitState state) throws InvalidProtocolBufferException{ - // Returns a branch number + public SeldonMessage route(SeldonMessage input, PredictiveUnitState state) throws InvalidProtocolBufferException{ + // Returns a branch number in SeldonMessage if (predictorConfig.hasMethod(PredictiveUnitMethod.ROUTE, state)){ - SeldonMessage routerReturn = internalPredictionService.route(input, state); - return getBranchIndex(routerReturn, state); + return internalPredictionService.route(input, state); } - return -1; + + // Return default routing + return null; } public void doSendFeedback(Feedback feedback, PredictiveUnitState state) throws InvalidProtocolBufferException{ @@ -245,6 +285,31 @@ protected void doStoreFeedbackMetrics(Feedback feedback, PredictiveUnitState sta Counter.builder("seldon_api_model_feedback").tags(tagsProvider.getModelMetrics(state)).register(Metrics.globalRegistry).increment(); } + private void addCustomMetrics(List metrics, PredictiveUnitState state) + { + logger.debug("Add metrics"); + for(Metric metric : metrics) + { + switch(metric.getType()) + { + case COUNTER: + logger.debug("Adding counter {} for {}",metric.getKey(),state.name); + Counter.builder(metric.getKey()).tags(tagsProvider.getModelMetrics(state)).register(Metrics.globalRegistry).increment(metric.getValue()); + break; + case GAUGE: + logger.debug("Adding gauge {} for {}",metric.getKey(),state.name); + customMetricsManager.get(state, metric).set(metric.getValue()); + break; + case TIMER: + logger.debug("Adding timer {} for {}",metric.getKey(),state.name); + Timer.builder(metric.getKey()).tags(tagsProvider.getModelMetrics(state)).register(Metrics.globalRegistry).record((long) metric.getValue(), TimeUnit.MILLISECONDS); + break; + case UNRECOGNIZED: + break; + } + } + } + private void sanityCheckRouting(Integer branchIndex, PredictiveUnitState state){ if (branchIndex < -1 | branchIndex >= state.children.size()){ throw new APIException( @@ -258,12 +323,14 @@ private SeldonMessage mergeMeta(SeldonMessage message, List messa for (SeldonMessage originalMessage : messages){ metaBuilder.putAllTags(originalMessage.getMeta().getTagsMap()); } + metaBuilder.clearMetrics(); return SeldonMessage.newBuilder(message).setMeta(metaBuilder).build(); } private SeldonMessage mergeMeta(SeldonMessage message, Meta meta) { Meta.Builder metaBuilder = Meta.newBuilder(message.getMeta()); metaBuilder.putAllTags(meta.getTagsMap()); + metaBuilder.clearMetrics(); return SeldonMessage.newBuilder(message).setMeta(metaBuilder).build(); } diff --git a/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitImpl.java b/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitImpl.java index 8a83478bff..1804d432a6 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitImpl.java +++ b/engine/src/main/java/io/seldon/engine/predictors/PredictiveUnitImpl.java @@ -19,8 +19,10 @@ import com.google.protobuf.InvalidProtocolBufferException; +import io.seldon.protos.PredictionProtos.DefaultData; import io.seldon.protos.PredictionProtos.Feedback; import io.seldon.protos.PredictionProtos.SeldonMessage; +import io.seldon.protos.PredictionProtos.Tensor; public abstract class PredictiveUnitImpl { @@ -36,8 +38,8 @@ public SeldonMessage aggregate(List outputs, PredictiveUnitState return outputs.get(0); } - public int route(SeldonMessage input, PredictiveUnitState state) throws InvalidProtocolBufferException{ - return -1; + public SeldonMessage route(SeldonMessage input, PredictiveUnitState state) throws InvalidProtocolBufferException{ + return SeldonMessage.newBuilder().setData(DefaultData.newBuilder().setTensor(Tensor.newBuilder().addValues(-1).addShape(1).addShape(1))).build(); } public void doSendFeedback(Feedback feedback, PredictiveUnitState state) throws InvalidProtocolBufferException{ diff --git a/engine/src/main/java/io/seldon/engine/predictors/RandomABTestUnit.java b/engine/src/main/java/io/seldon/engine/predictors/RandomABTestUnit.java index 1559b89c07..93681ff30f 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/RandomABTestUnit.java +++ b/engine/src/main/java/io/seldon/engine/predictors/RandomABTestUnit.java @@ -20,7 +20,9 @@ import org.springframework.stereotype.Component; import io.seldon.engine.exception.APIException; +import io.seldon.protos.PredictionProtos.DefaultData; import io.seldon.protos.PredictionProtos.SeldonMessage; +import io.seldon.protos.PredictionProtos.Tensor; @Component @@ -31,7 +33,7 @@ public class RandomABTestUnit extends PredictiveUnitImpl{ public RandomABTestUnit() {} @Override - public int route(SeldonMessage input, PredictiveUnitState state){ + public SeldonMessage route(SeldonMessage input, PredictiveUnitState state){ @SuppressWarnings("unchecked") PredictiveUnitParameter parameter = (PredictiveUnitParameter) state.parameters.get("ratioA"); @@ -49,10 +51,10 @@ public int route(SeldonMessage input, PredictiveUnitState state){ //FIXME Possible bug : keySet is not ordered as per the definition of the AB test if (comparator<=ratioA){ // We select model A - return 0; + return SeldonMessage.newBuilder().setData(DefaultData.newBuilder().setTensor(Tensor.newBuilder().addValues(0).addShape(1).addShape(1))).build(); } else{ - return 1; + return SeldonMessage.newBuilder().setData(DefaultData.newBuilder().setTensor(Tensor.newBuilder().addValues(1).addShape(1).addShape(1))).build(); } } } diff --git a/engine/src/main/java/io/seldon/engine/predictors/SimpleModelUnit.java b/engine/src/main/java/io/seldon/engine/predictors/SimpleModelUnit.java index f04331703a..87202ecb99 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/SimpleModelUnit.java +++ b/engine/src/main/java/io/seldon/engine/predictors/SimpleModelUnit.java @@ -22,6 +22,8 @@ import io.seldon.protos.PredictionProtos.DefaultData; import io.seldon.protos.PredictionProtos.SeldonMessage; import io.seldon.protos.PredictionProtos.Meta; +import io.seldon.protos.PredictionProtos.Metric; +import io.seldon.protos.PredictionProtos.Metric.MetricType; import io.seldon.protos.PredictionProtos.Status; import io.seldon.protos.PredictionProtos.Tensor; @@ -37,7 +39,10 @@ public SimpleModelUnit() {} public SeldonMessage transformInput(SeldonMessage input, PredictiveUnitState state){ SeldonMessage output = SeldonMessage.newBuilder() .setStatus(Status.newBuilder().setStatus(Status.StatusFlag.SUCCESS).build()) - .setMeta(Meta.newBuilder())//.addModel(state.id)) + .setMeta(Meta.newBuilder() + .addMetrics(Metric.newBuilder().setKey("mymetric_counter").setType(MetricType.COUNTER).setValue(1)) + .addMetrics(Metric.newBuilder().setKey("mymetric_gauge").setType(MetricType.GAUGE).setValue(100)) + .addMetrics(Metric.newBuilder().setKey("mymetric_timer").setType(MetricType.TIMER).setValue(22.1F))) .setData(DefaultData.newBuilder().addAllNames(Arrays.asList(classes)) .setTensor(Tensor.newBuilder().addShape(1).addShape(values.length) .addAllValues(Arrays.asList(values)))).build(); diff --git a/engine/src/main/java/io/seldon/engine/predictors/SimpleRouterUnit.java b/engine/src/main/java/io/seldon/engine/predictors/SimpleRouterUnit.java index 4cd46ae4ef..cdde8b01ed 100644 --- a/engine/src/main/java/io/seldon/engine/predictors/SimpleRouterUnit.java +++ b/engine/src/main/java/io/seldon/engine/predictors/SimpleRouterUnit.java @@ -17,8 +17,9 @@ import org.springframework.stereotype.Component; -import io.seldon.protos.PredictionProtos.Feedback; +import io.seldon.protos.PredictionProtos.DefaultData; import io.seldon.protos.PredictionProtos.SeldonMessage; +import io.seldon.protos.PredictionProtos.Tensor; @Component public class SimpleRouterUnit extends PredictiveUnitImpl { @@ -26,7 +27,7 @@ public class SimpleRouterUnit extends PredictiveUnitImpl { public SimpleRouterUnit() {} @Override - public int route(SeldonMessage input, PredictiveUnitState state){ - return 0; + public SeldonMessage route(SeldonMessage input, PredictiveUnitState state){ + return SeldonMessage.newBuilder().setData(DefaultData.newBuilder().setTensor(Tensor.newBuilder().addValues(0).addShape(1).addShape(1))).build(); } } diff --git a/engine/src/main/java/io/seldon/engine/service/InternalPredictionService.java b/engine/src/main/java/io/seldon/engine/service/InternalPredictionService.java index 69bc9f8c96..bc7c2d2cc0 100644 --- a/engine/src/main/java/io/seldon/engine/service/InternalPredictionService.java +++ b/engine/src/main/java/io/seldon/engine/service/InternalPredictionService.java @@ -390,4 +390,12 @@ private SeldonMessage queryREST(String path, String dataString, PredictiveUnitSt } } + /** + * Used only for testing. Should be replaced by better methods that use Spring and Mockito to create a Mock RestTemplate for testing + * @param predictorSpec + */ + public void setRestTemplate(RestTemplate restTemplate) { // FIXME + this.restTemplate = restTemplate; + } + } diff --git a/engine/src/test/java/io/seldon/engine/api/rest/TestRandomABTest.java b/engine/src/test/java/io/seldon/engine/api/rest/TestRandomABTest.java new file mode 100644 index 0000000000..75b9ca5728 --- /dev/null +++ b/engine/src/test/java/io/seldon/engine/api/rest/TestRandomABTest.java @@ -0,0 +1,159 @@ +package io.seldon.engine.api.rest; + +import java.io.IOException; +import java.net.URI; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; + +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Matchers; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.context.embedded.LocalServerPort; +import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.http.HttpEntity; +import org.springframework.http.HttpStatus; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.test.context.TestPropertySource; +import org.springframework.test.context.junit4.SpringRunner; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.test.web.servlet.MvcResult; +import org.springframework.test.web.servlet.request.MockMvcRequestBuilders; +import org.springframework.test.web.servlet.setup.MockMvcBuilders; +import org.springframework.util.MultiValueMap; +import org.springframework.web.client.RestTemplate; +import org.springframework.web.context.WebApplicationContext; + +import com.google.protobuf.InvalidProtocolBufferException; +import com.google.protobuf.Message; + +import io.kubernetes.client.proto.IntStr.IntOrString; +import io.kubernetes.client.proto.Meta.Time; +import io.kubernetes.client.proto.Meta.Timestamp; +import io.kubernetes.client.proto.Resource.Quantity; +import io.seldon.engine.pb.IntOrStringUtils; +import io.seldon.engine.pb.JsonFormat; +import io.seldon.engine.pb.QuantityUtils; +import io.seldon.engine.pb.TimeUtils; +import io.seldon.engine.predictors.EnginePredictor; +import io.seldon.engine.service.InternalPredictionService; +import io.seldon.protos.DeploymentProtos.PredictorSpec; +import io.seldon.protos.PredictionProtos.SeldonMessage; + +@RunWith(SpringRunner.class) +@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT) +@AutoConfigureMockMvc +@TestPropertySource(properties = { + "management.security.enabled=false", + }) +public class TestRandomABTest { + + protected String readFile(String path, Charset encoding) + throws IOException + { + byte[] encoded = Files.readAllBytes(Paths.get(path)); + return new String(encoded, encoding); + } + + private void updateMessageBuilderFromJson(T messageBuilder, String json) throws InvalidProtocolBufferException { + JsonFormat.parser().ignoringUnknownFields() + .usingTypeParser(IntOrString.getDescriptor().getFullName(), new IntOrStringUtils.IntOrStringParser()) + .usingTypeParser(Quantity.getDescriptor().getFullName(), new QuantityUtils.QuantityParser()) + .usingTypeParser(Time.getDescriptor().getFullName(), new TimeUtils.TimeParser()) + .usingTypeParser(Timestamp.getDescriptor().getFullName(), new TimeUtils.TimeParser()) + .merge(json, messageBuilder); + } + + @Autowired + private WebApplicationContext context; + + @Autowired + EnginePredictor enginePredictor; + + + //@Autowired + private MockMvc mvc; + + @Autowired + RestClientController restController; + + @Before + public void setup() throws Exception { + mvc = MockMvcBuilders + .webAppContextSetup(context) + .build(); + } + + @LocalServerPort + private int port; + + @Mock + private RestTemplate restTemplate; + + @Autowired + private InternalPredictionService internalPredictionService; + + + + @Test + public void testModelMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/abtest.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/response_with_metrics.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + int routeACount = 0; + int routeBCount = 1; + + for(int i=0;i<100;i++) + { + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + Assert.assertTrue(seldonMessage.getMeta().getRoutingMap().get("abtest") >= 0); + if (seldonMessage.getMeta().getRoutingMap().get("abtest") == 0) + routeACount++; + else + routeBCount++; + } + double split = routeACount /(double)(routeACount + routeBCount); + System.out.println(routeACount); + System.out.println(routeBCount); + Assert.assertEquals(0.5, split,0.2); + + + + } + +} diff --git a/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientController.java b/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientController.java index 83e7ba4161..b674a09534 100644 --- a/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientController.java +++ b/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientController.java @@ -25,6 +25,7 @@ import org.springframework.boot.test.context.SpringBootTest; import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; import org.springframework.http.MediaType; +import org.springframework.jmx.support.MetricType; import org.springframework.test.context.junit4.SpringRunner; import org.springframework.test.web.servlet.MockMvc; import org.springframework.test.web.servlet.MvcResult; @@ -32,6 +33,9 @@ import org.springframework.test.web.servlet.setup.MockMvcBuilders; import org.springframework.web.context.WebApplicationContext; +import io.seldon.engine.pb.ProtoBufUtils; +import io.seldon.protos.PredictionProtos.SeldonMessage; + @RunWith(SpringRunner.class) @SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT) @AutoConfigureMockMvc @@ -68,7 +72,7 @@ public void testPing() throws Exception @Test - public void testPredict_11dim() throws Exception + public void testPredict_11dim_ndarry() throws Exception { final String predictJson = "{" + "\"request\": {" + @@ -85,7 +89,7 @@ public void testPredict_11dim() throws Exception } @Test - public void testPredict_21dim() throws Exception + public void testPredict_21dim_ndarry() throws Exception { final String predictJson = "{" + "\"request\": {" + @@ -99,5 +103,36 @@ public void testPredict_21dim() throws Exception String response = res.getResponse().getContentAsString(); System.out.println(response); Assert.assertEquals(200, res.getResponse().getStatus()); + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + ProtoBufUtils.updateMessageBuilderFromJson(builder, response ); + SeldonMessage seldonMessage = builder.build(); + Assert.assertEquals(3, seldonMessage.getMeta().getMetricsCount()); + Assert.assertEquals("COUNTER", seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals("GAUGE", seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals("TIMER", seldonMessage.getMeta().getMetrics(2).getType().toString()); + } + + @Test + public void testPredict_21dim_tensor() throws Exception + { + final String predictJson = "{" + + "\"request\": {" + + "\"tensor\": {\"shape\":[2,1],\"values\":[1.0,2.0]}}" + + "}"; + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + ProtoBufUtils.updateMessageBuilderFromJson(builder, response ); + SeldonMessage seldonMessage = builder.build(); + Assert.assertEquals(3, seldonMessage.getMeta().getMetricsCount()); + Assert.assertEquals("COUNTER", seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals("GAUGE", seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals("TIMER", seldonMessage.getMeta().getMetrics(2).getType().toString()); } } diff --git a/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientControllerExternalGraphs.java b/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientControllerExternalGraphs.java new file mode 100644 index 0000000000..6d9665fb62 --- /dev/null +++ b/engine/src/test/java/io/seldon/engine/api/rest/TestRestClientControllerExternalGraphs.java @@ -0,0 +1,385 @@ +package io.seldon.engine.api.rest; + +import java.io.IOException; +import java.net.URI; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; + +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.InjectMocks; +import org.mockito.Matchers; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.context.embedded.LocalServerPort; +import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.http.HttpEntity; +import org.springframework.http.HttpStatus; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.test.context.TestPropertySource; +import org.springframework.test.context.junit4.SpringRunner; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.test.web.servlet.MvcResult; +import org.springframework.test.web.servlet.request.MockMvcRequestBuilders; +import org.springframework.test.web.servlet.setup.MockMvcBuilders; +import org.springframework.util.MultiValueMap; +import org.springframework.web.client.RestTemplate; +import org.springframework.web.context.WebApplicationContext; + +import com.google.protobuf.InvalidProtocolBufferException; +import com.google.protobuf.Message; + +import io.kubernetes.client.proto.IntStr.IntOrString; +import io.kubernetes.client.proto.Meta.Time; +import io.kubernetes.client.proto.Meta.Timestamp; +import io.kubernetes.client.proto.Resource.Quantity; +import io.seldon.engine.pb.IntOrStringUtils; +import io.seldon.engine.pb.JsonFormat; +import io.seldon.engine.pb.QuantityUtils; +import io.seldon.engine.pb.TimeUtils; +import io.seldon.engine.predictors.EnginePredictor; +import io.seldon.engine.service.InternalPredictionService; +import io.seldon.protos.DeploymentProtos.PredictorSpec; +import io.seldon.protos.PredictionProtos.SeldonMessage; + +@RunWith(SpringRunner.class) +@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT) +@AutoConfigureMockMvc +@TestPropertySource(properties = { + "management.security.enabled=false", + }) +public class TestRestClientControllerExternalGraphs { + + protected String readFile(String path, Charset encoding) + throws IOException + { + byte[] encoded = Files.readAllBytes(Paths.get(path)); + return new String(encoded, encoding); + } + + private void updateMessageBuilderFromJson(T messageBuilder, String json) throws InvalidProtocolBufferException { + JsonFormat.parser().ignoringUnknownFields() + .usingTypeParser(IntOrString.getDescriptor().getFullName(), new IntOrStringUtils.IntOrStringParser()) + .usingTypeParser(Quantity.getDescriptor().getFullName(), new QuantityUtils.QuantityParser()) + .usingTypeParser(Time.getDescriptor().getFullName(), new TimeUtils.TimeParser()) + .usingTypeParser(Timestamp.getDescriptor().getFullName(), new TimeUtils.TimeParser()) + .merge(json, messageBuilder); + } + + @Autowired + private WebApplicationContext context; + + @Autowired + EnginePredictor enginePredictor; + + + //@Autowired + private MockMvc mvc; + + @Autowired + RestClientController restController; + + @Before + public void setup() throws Exception { + mvc = MockMvcBuilders + .webAppContextSetup(context) + .build(); + } + + @LocalServerPort + private int port; + + @Mock + private RestTemplate restTemplate; + + @Autowired + private InternalPredictionService internalPredictionService; + + + + @Test + public void testModelMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/model_simple.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/response_with_metrics.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + // Check for returned metrics + Assert.assertEquals("COUNTER",seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(0).getValue(),0.0); + Assert.assertEquals("mycounter",seldonMessage.getMeta().getMetrics(0).getKey()); + + Assert.assertEquals("GAUGE",seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals(22.0F,seldonMessage.getMeta().getMetrics(1).getValue(),0.0); + Assert.assertEquals("mygauge",seldonMessage.getMeta().getMetrics(1).getKey()); + + Assert.assertEquals("TIMER",seldonMessage.getMeta().getMetrics(2).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(2).getValue(),0.0); + Assert.assertEquals("mytimer",seldonMessage.getMeta().getMetrics(2).getKey()); + + // Check prometheus endpoint for metric + MvcResult res2 = mvc.perform(MockMvcRequestBuilders.get("/prometheus")).andReturn(); + Assert.assertEquals(200, res2.getResponse().getStatus()); + response = res2.getResponse().getContentAsString(); + Assert.assertTrue(response.indexOf("mycounter_total{deployment_name=\"None\",model_image=\"seldonio/mean_classifier\",model_name=\"mean-classifier\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + Assert.assertTrue(response.indexOf("mytimer_duration_seconds_count{deployment_name=\"None\",model_image=\"seldonio/mean_classifier\",model_name=\"mean-classifier\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + System.out.println(response); + } + + + @Test + public void testInputTransformInputMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/transformer_simple.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/response_with_metrics.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + // Check for returned metrics + Assert.assertEquals("COUNTER",seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(0).getValue(),0.0); + Assert.assertEquals("mycounter",seldonMessage.getMeta().getMetrics(0).getKey()); + + Assert.assertEquals("GAUGE",seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals(22.0F,seldonMessage.getMeta().getMetrics(1).getValue(),0.0); + Assert.assertEquals("mygauge",seldonMessage.getMeta().getMetrics(1).getKey()); + + Assert.assertEquals("TIMER",seldonMessage.getMeta().getMetrics(2).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(2).getValue(),0.0); + Assert.assertEquals("mytimer",seldonMessage.getMeta().getMetrics(2).getKey()); + + // Check prometheus endpoint for metric + MvcResult res2 = mvc.perform(MockMvcRequestBuilders.get("/prometheus")).andReturn(); + Assert.assertEquals(200, res2.getResponse().getStatus()); + response = res2.getResponse().getContentAsString(); + Assert.assertTrue(response.indexOf("mycounter_total{deployment_name=\"None\",model_image=\"seldonio/transformer\",model_name=\"transformer\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + Assert.assertTrue(response.indexOf("mytimer_duration_seconds_count{deployment_name=\"None\",model_image=\"seldonio/transformer\",model_name=\"transformer\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + System.out.println(response); + } + + + @Test + public void testTransformOutputMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/transform_output_simple.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/response_with_metrics.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + // Check for returned metrics + Assert.assertEquals("COUNTER",seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(0).getValue(),0.0); + Assert.assertEquals("mycounter",seldonMessage.getMeta().getMetrics(0).getKey()); + + Assert.assertEquals("GAUGE",seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals(22.0F,seldonMessage.getMeta().getMetrics(1).getValue(),0.0); + Assert.assertEquals("mygauge",seldonMessage.getMeta().getMetrics(1).getKey()); + + Assert.assertEquals("TIMER",seldonMessage.getMeta().getMetrics(2).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(2).getValue(),0.0); + Assert.assertEquals("mytimer",seldonMessage.getMeta().getMetrics(2).getKey()); + + // Check prometheus endpoint for metric + MvcResult res2 = mvc.perform(MockMvcRequestBuilders.get("/prometheus")).andReturn(); + Assert.assertEquals(200, res2.getResponse().getStatus()); + response = res2.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertTrue(response.indexOf("mycounter_total{deployment_name=\"None\",model_image=\"seldonio/transformer\",model_name=\"transform_output\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + Assert.assertTrue(response.indexOf("mytimer_duration_seconds_count{deployment_name=\"None\",model_image=\"seldonio/transformer\",model_name=\"transform_output\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + + } + + + @Test + public void testRouterMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/router_simple.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/router_response.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + // Check for returned metrics + Assert.assertEquals("COUNTER",seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(0).getValue(),0.0); + Assert.assertEquals("mycounter",seldonMessage.getMeta().getMetrics(0).getKey()); + + Assert.assertEquals("GAUGE",seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals(22.0F,seldonMessage.getMeta().getMetrics(1).getValue(),0.0); + Assert.assertEquals("mygauge",seldonMessage.getMeta().getMetrics(1).getKey()); + + Assert.assertEquals("TIMER",seldonMessage.getMeta().getMetrics(2).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(2).getValue(),0.0); + Assert.assertEquals("mytimer",seldonMessage.getMeta().getMetrics(2).getKey()); + + // Check prometheus endpoint for metric + MvcResult res2 = mvc.perform(MockMvcRequestBuilders.get("/prometheus")).andReturn(); + Assert.assertEquals(200, res2.getResponse().getStatus()); + response = res2.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertTrue(response.indexOf("mycounter_total{deployment_name=\"None\",model_image=\"seldonio/router\",model_name=\"router\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + Assert.assertTrue(response.indexOf("mytimer_duration_seconds_count{deployment_name=\"None\",model_image=\"seldonio/router\",model_name=\"router\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + + } + + + @Test + public void testCombinerMetrics() throws Exception + { + String jsonStr = readFile("src/test/resources/combiner_simple.json",StandardCharsets.UTF_8); + String responseStr = readFile("src/test/resources/response_with_metrics.json",StandardCharsets.UTF_8); + PredictorSpec.Builder PredictorSpecBuilder = PredictorSpec.newBuilder(); + updateMessageBuilderFromJson(PredictorSpecBuilder, jsonStr); + PredictorSpec predictorSpec = PredictorSpecBuilder.build(); + final String predictJson = "{" + + "\"request\": {" + + "\"ndarray\": [[1.0]]}" + + "}"; + enginePredictor.setPredictorSpec(predictorSpec); + + + ResponseEntity httpResponse = new ResponseEntity(responseStr, null, HttpStatus.OK); + Mockito.when(restTemplate.postForEntity(Matchers.any(), Matchers.>>any(), Matchers.>any())) + .thenReturn(httpResponse); + internalPredictionService.setRestTemplate(restTemplate); + + MvcResult res = mvc.perform(MockMvcRequestBuilders.post("/api/v0.1/predictions") + .accept(MediaType.APPLICATION_JSON_UTF8) + .content(predictJson) + .contentType(MediaType.APPLICATION_JSON_UTF8)).andReturn(); + String response = res.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertEquals(200, res.getResponse().getStatus()); + + SeldonMessage.Builder builder = SeldonMessage.newBuilder(); + JsonFormat.parser().ignoringUnknownFields().merge(response, builder); + SeldonMessage seldonMessage = builder.build(); + + // Check for returned metrics + Assert.assertEquals("COUNTER",seldonMessage.getMeta().getMetrics(0).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(0).getValue(),0.0); + Assert.assertEquals("mycounter",seldonMessage.getMeta().getMetrics(0).getKey()); + + Assert.assertEquals("GAUGE",seldonMessage.getMeta().getMetrics(1).getType().toString()); + Assert.assertEquals(22.0F,seldonMessage.getMeta().getMetrics(1).getValue(),0.0); + Assert.assertEquals("mygauge",seldonMessage.getMeta().getMetrics(1).getKey()); + + Assert.assertEquals("TIMER",seldonMessage.getMeta().getMetrics(2).getType().toString()); + Assert.assertEquals(1.0F,seldonMessage.getMeta().getMetrics(2).getValue(),0.0); + Assert.assertEquals("mytimer",seldonMessage.getMeta().getMetrics(2).getKey()); + + // Check prometheus endpoint for metric + MvcResult res2 = mvc.perform(MockMvcRequestBuilders.get("/prometheus")).andReturn(); + Assert.assertEquals(200, res2.getResponse().getStatus()); + response = res2.getResponse().getContentAsString(); + System.out.println(response); + Assert.assertTrue(response.indexOf("mycounter_total{deployment_name=\"None\",model_image=\"seldonio/combiner\",model_name=\"combiner\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + Assert.assertTrue(response.indexOf("mytimer_duration_seconds_count{deployment_name=\"None\",model_image=\"seldonio/combiner\",model_name=\"combiner\",model_version=\"0.6\",predictor_name=\"fx-market-predictor\",predictor_version=\"unknown\",} 1.0")>-1); + + } + +} diff --git a/engine/src/test/java/io/seldon/engine/pb/TestPredictionProto.java b/engine/src/test/java/io/seldon/engine/pb/TestPredictionProto.java index 46034aa1b7..1bbb138579 100644 --- a/engine/src/test/java/io/seldon/engine/pb/TestPredictionProto.java +++ b/engine/src/test/java/io/seldon/engine/pb/TestPredictionProto.java @@ -49,7 +49,7 @@ public void parse_json_extra_fields() throws InvalidProtocolBufferException @Test public void parse_custom_json() throws InvalidProtocolBufferException { - String json = "{\"request\":{\"ndarray\":[[1.0,2.0],[3.0,4.0]]}}"; + String json = "{\"data\":{\"ndarray\":[[1.0,2.0],[3.0,4.0]]}}"; SeldonMessage.Builder builder = SeldonMessage.newBuilder(); ProtoBufUtils.updateMessageBuilderFromJson(builder, json); SeldonMessage request = builder.build(); @@ -64,7 +64,7 @@ public void parse_custom_json() throws InvalidProtocolBufferException @Test public void parse_tags_array() throws InvalidProtocolBufferException { - String json = "{\"meta\":{\"tags\":{\"user\":[\"a\",\"b\"],\"gender\":\"female\"}},\"request\":{\"ndarray\":[[1.0,2.0],[3.0,4.0]]}}"; + String json = "{\"meta\":{\"tags\":{\"user\":[\"a\",\"b\"],\"gender\":\"female\"}},\"data\":{\"ndarray\":[[1.0,2.0],[3.0,4.0]]}}"; SeldonMessage.Builder builder = SeldonMessage.newBuilder(); ProtoBufUtils.updateMessageBuilderFromJson(builder, json); SeldonMessage request = builder.build(); diff --git a/engine/src/test/java/io/seldon/engine/predictors/RandomABTestUnitInternalTest.java b/engine/src/test/java/io/seldon/engine/predictors/RandomABTestUnitInternalTest.java index f4509ac117..5bf48175c8 100644 --- a/engine/src/test/java/io/seldon/engine/predictors/RandomABTestUnitInternalTest.java +++ b/engine/src/test/java/io/seldon/engine/predictors/RandomABTestUnitInternalTest.java @@ -21,6 +21,7 @@ import org.junit.Assert; import org.junit.Test; +import org.ojalgo.matrix.PrimitiveMatrix; import com.google.protobuf.InvalidProtocolBufferException; @@ -30,6 +31,12 @@ public class RandomABTestUnitInternalTest { + private int getBranchIndex(SeldonMessage routerReturn) { + PrimitiveMatrix dataArray = PredictorUtils.getOJMatrix(routerReturn.getData()); + return dataArray.get(0).intValue(); + } + + @Test public void simpleCase() throws NoSuchMethodException, SecurityException, IllegalAccessException, IllegalArgumentException, InvalidProtocolBufferException { @@ -50,15 +57,18 @@ public void simpleCase() throws NoSuchMethodException, SecurityException, Illega PredictiveUnitImpl predictiveUnit = new RandomABTestUnit(); // The following values are from random seed 1337 - int routing1 = (int) predictiveUnit.route(request, state); + SeldonMessage msg1 = predictiveUnit.route(request, state); + int routing1 = getBranchIndex(msg1); Assert.assertEquals(1,routing1); - int routing2 = (int) predictiveUnit.route(request, state); + SeldonMessage msg2 = predictiveUnit.route(request, state); + int routing2 = (int) getBranchIndex(msg2); Assert.assertEquals(0,routing2); - int routing3 = (int) predictiveUnit.route(request,state); + SeldonMessage msg3 = predictiveUnit.route(request,state); + int routing3 = (int) getBranchIndex(msg3); Assert.assertEquals(1,routing3); } diff --git a/engine/src/test/resources/abtest.json b/engine/src/test/resources/abtest.json new file mode 100644 index 0000000000..9b848c202f --- /dev/null +++ b/engine/src/test/resources/abtest.json @@ -0,0 +1,82 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/model1:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "model1", + "resources": { + "requests": { + "memory": "1Mi" + } + } + }, + { + "image": "seldonio/model2:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "model2", + "resources": { + "requests": { + "memory": "1Mi" + } + } + }, + { + "image": "seldonio/router:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "router", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + { + "name": "model1", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9001 + }, + "type": "MODEL" + }, + { + "name": "model2", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9002 + }, + "type": "MODEL" + } + ], + "name": "abtest", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "implementation": "RANDOM_ABTEST", + "parameters": [ + { + "name": "ratioA", + "value": "0.5", + "type": "FLOAT" + } + ] + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} diff --git a/engine/src/test/resources/combiner_simple.json b/engine/src/test/resources/combiner_simple.json new file mode 100644 index 0000000000..976e4def2e --- /dev/null +++ b/engine/src/test/resources/combiner_simple.json @@ -0,0 +1,56 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/model:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "model", + "resources": { + "requests": { + "memory": "1Mi" + } + } + }, + { + "image": "seldonio/combiner:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "combiner", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + { + "name": "model", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9001 + }, + "type": "MODEL" + } + ], + "name": "combiner", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "type": "COMBINER" + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} \ No newline at end of file diff --git a/engine/src/test/resources/model_simple.json b/engine/src/test/resources/model_simple.json new file mode 100644 index 0000000000..cdae8f7b87 --- /dev/null +++ b/engine/src/test/resources/model_simple.json @@ -0,0 +1,37 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/mean_classifier:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "mean-classifier", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + ], + "name": "mean-classifier", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "type": "MODEL" + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} \ No newline at end of file diff --git a/engine/src/test/resources/response_with_metrics.json b/engine/src/test/resources/response_with_metrics.json new file mode 100644 index 0000000000..90cf2a937a --- /dev/null +++ b/engine/src/test/resources/response_with_metrics.json @@ -0,0 +1,29 @@ +{ + "meta": { + "metrics": [ + { + "type": "COUNTER", + "key": "mycounter", + "value": 1.0 + }, + { + "type": "GAUGE", + "key": "mygauge", + "value": 22.0 + }, + { + "type": "TIMER", + "key": "mytimer", + "value": 1.0 + } + ] + }, + "data": { + "ndarray": [ + [ + 1, + 2 + ] + ] + } +} diff --git a/engine/src/test/resources/router_response.json b/engine/src/test/resources/router_response.json new file mode 100644 index 0000000000..ad7e41b959 --- /dev/null +++ b/engine/src/test/resources/router_response.json @@ -0,0 +1,28 @@ +{ + "meta": { + "metrics": [ + { + "type": "COUNTER", + "key": "mycounter", + "value": 1.0 + }, + { + "type": "GAUGE", + "key": "mygauge", + "value": 22.0 + }, + { + "type": "TIMER", + "key": "mytimer", + "value": 1.0 + } + ] + }, + "data": { + "ndarray": [ + [ + 0 + ] + ] + } +} diff --git a/engine/src/test/resources/router_simple.json b/engine/src/test/resources/router_simple.json new file mode 100644 index 0000000000..db88c8c9f9 --- /dev/null +++ b/engine/src/test/resources/router_simple.json @@ -0,0 +1,56 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/model:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "model", + "resources": { + "requests": { + "memory": "1Mi" + } + } + }, + { + "image": "seldonio/router:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "router", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + { + "name": "model", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9001 + }, + "type": "MODEL" + } + ], + "name": "router", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "type": "ROUTER" + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} \ No newline at end of file diff --git a/engine/src/test/resources/transform_output_simple.json b/engine/src/test/resources/transform_output_simple.json new file mode 100644 index 0000000000..2ecf13f6fa --- /dev/null +++ b/engine/src/test/resources/transform_output_simple.json @@ -0,0 +1,56 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/model:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "model", + "resources": { + "requests": { + "memory": "1Mi" + } + } + }, + { + "image": "seldonio/transformer:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "transform_output", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + { + "name": "model", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9001 + }, + "type": "MODEL" + } + ], + "name": "transform_output", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "type": "OUTPUT_TRANSFORMER" + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} \ No newline at end of file diff --git a/engine/src/test/resources/transformer_simple.json b/engine/src/test/resources/transformer_simple.json new file mode 100644 index 0000000000..5665c580b8 --- /dev/null +++ b/engine/src/test/resources/transformer_simple.json @@ -0,0 +1,37 @@ +{ + "componentSpecs": [ + { + "spec": { + "containers": [ + { + "image": "seldonio/transformer:0.6", + "imagePullPolicy": "IfNotPresent", + "name": "transformer", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + } + ], + "graph": { + "children": [ + ], + "name": "transformer", + "endpoint": { + "type": "REST", + "service_host": "somehost", + "service_port": 9000 + }, + "type": "TRANSFORMER" + }, + "name": "fx-market-predictor", + "replicas": 1, + "annotations": { + "predictor_version": "v1" + } +} \ No newline at end of file diff --git a/examples/models/template_model_with_metrics/ModelWithMetrics.py b/examples/models/template_model_with_metrics/ModelWithMetrics.py new file mode 100644 index 0000000000..c5bc063f20 --- /dev/null +++ b/examples/models/template_model_with_metrics/ModelWithMetrics.py @@ -0,0 +1,17 @@ + +class ModelWithMetrics(object): + + def __init__(self): + print("Initialising") + + def predict(self,X,features_names): + print("Predict called") + return X + + def metrics(self): + return [ + {"type":"COUNTER","key":"mycounter","value":1}, # a counter which will increase by the given value + {"type":"GAUGE","key":"mygauge","value":100}, # a gauge which will be set to given value + {"type":"TIMER","key":"mytimer","value":20.2}, # a timer which will add sum and count metrics - assumed millisecs + ] + diff --git a/examples/models/template_model_with_metrics/contract.json b/examples/models/template_model_with_metrics/contract.json new file mode 100644 index 0000000000..ab7102d7e5 --- /dev/null +++ b/examples/models/template_model_with_metrics/contract.json @@ -0,0 +1,39 @@ +{ + "features":[ + { + "name":"sepal_length", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[4,8] + }, + { + "name":"sepal_width", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[2,5] + }, + { + "name":"petal_length", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[1,10] + }, + { + "name":"petal_width", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[0,3] + } + ], + "targets":[ + { + "name":"class", + "dtype":"FLOAT", + "ftype":"continuous", + "range":[0,1], + "repeat":3 + } + ] +} + + diff --git a/examples/models/template_model_with_metrics/deployment-grpc.json b/examples/models/template_model_with_metrics/deployment-grpc.json new file mode 100644 index 0000000000..05c91f9a68 --- /dev/null +++ b/examples/models/template_model_with_metrics/deployment-grpc.json @@ -0,0 +1,46 @@ +{ + "apiVersion": "machinelearning.seldon.io/v1alpha2", + "kind": "SeldonDeployment", + "metadata": { + "labels": { + "app": "seldon" + }, + "name": "mymodel" + }, + "spec": { + "name": "mymodel", + "oauth_key": "oauth-key", + "oauth_secret": "oauth-secret", + "predictors": [ + { + "componentSpecs": [{ + "spec": { + "containers": [ + { + "image": "model-with-metrics-grpc:0.1", + "imagePullPolicy": "IfNotPresent", + "name": "complex-model", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + }], + "graph": { + "children": [], + "name": "complex-model", + "endpoint": { + "type" : "GRPC" + }, + "type": "MODEL" + }, + "name": "mymodel", + "replicas": 1 + } + ] + } +} diff --git a/examples/models/template_model_with_metrics/deployment-rest.json b/examples/models/template_model_with_metrics/deployment-rest.json new file mode 100644 index 0000000000..541e993767 --- /dev/null +++ b/examples/models/template_model_with_metrics/deployment-rest.json @@ -0,0 +1,46 @@ +{ + "apiVersion": "machinelearning.seldon.io/v1alpha2", + "kind": "SeldonDeployment", + "metadata": { + "labels": { + "app": "seldon" + }, + "name": "mymodel" + }, + "spec": { + "name": "mymodel", + "oauth_key": "oauth-key", + "oauth_secret": "oauth-secret", + "predictors": [ + { + "componentSpecs": [{ + "spec": { + "containers": [ + { + "image": "model-with-metrics-rest:0.1", + "imagePullPolicy": "IfNotPresent", + "name": "complex-model", + "resources": { + "requests": { + "memory": "1Mi" + } + } + } + ], + "terminationGracePeriodSeconds": 20 + } + }], + "graph": { + "children": [], + "name": "complex-model", + "endpoint": { + "type" : "REST" + }, + "type": "MODEL" + }, + "name": "mymodel", + "replicas": 1 + } + ] + } +} diff --git a/examples/models/template_model_with_metrics/environment_grpc b/examples/models/template_model_with_metrics/environment_grpc new file mode 100644 index 0000000000..5cb5826fbe --- /dev/null +++ b/examples/models/template_model_with_metrics/environment_grpc @@ -0,0 +1,4 @@ +MODEL_NAME=ModelWithMetrics +API_TYPE=GRPC +SERVICE_TYPE=MODEL +PERSISTENCE=0 diff --git a/examples/models/template_model_with_metrics/environment_rest b/examples/models/template_model_with_metrics/environment_rest new file mode 100644 index 0000000000..dcfa81c73e --- /dev/null +++ b/examples/models/template_model_with_metrics/environment_rest @@ -0,0 +1,4 @@ +MODEL_NAME=ModelWithMetrics +API_TYPE=REST +SERVICE_TYPE=MODEL +PERSISTENCE=0 diff --git a/examples/models/template_model_with_metrics/modelWithMetrics.ipynb b/examples/models/template_model_with_metrics/modelWithMetrics.ipynb new file mode 100644 index 0000000000..adede8c300 --- /dev/null +++ b/examples/models/template_model_with_metrics/modelWithMetrics.ipynb @@ -0,0 +1,1143 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Model with Metrics\n", + "\n", + "Example testing a model with custom metrics.\n", + "\n", + "Metrics can be \n", + "\n", + " * A ```COUNTER``` : the returned value will increment the current value\n", + " * A ```GAUGE``` : the returned value will overwrite the current value\n", + " * A ```TIMER``` : a number of millisecs. Prometheus SUM and COUNT metrics will be created.\n", + " \n", + "You need to provide a list of dictionaries each with the following:\n", + "\n", + " * a ```type``` : COUNTER, GAUGE, or TIMER\n", + " * a ```key``` : a user defined key\n", + " * a ```value``` : a float value\n", + " \n", + "See example code below:\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mModelWithMetrics\u001b[39;49;00m(\u001b[36mobject\u001b[39;49;00m):\r\n", + "\r\n", + " \u001b[34mdef\u001b[39;49;00m \u001b[32m__init__\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m):\r\n", + " \u001b[34mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mInitialising\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n", + "\r\n", + " \u001b[34mdef\u001b[39;49;00m \u001b[32mpredict\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m,X,features_names):\r\n", + " \u001b[34mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mPredict called\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n", + " \u001b[34mreturn\u001b[39;49;00m X\r\n", + "\r\n", + " \u001b[34mdef\u001b[39;49;00m \u001b[32mmetrics\u001b[39;49;00m():\r\n", + " \u001b[34mreturn\u001b[39;49;00m [\r\n", + " {\u001b[33m\"\u001b[39;49;00m\u001b[33mtype\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mCOUNTER\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mkey\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mmycounter\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mvalue\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[34m1\u001b[39;49;00m}, \u001b[37m# a counter which will increase by the given value\u001b[39;49;00m\r\n", + " {\u001b[33m\"\u001b[39;49;00m\u001b[33mtype\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mGAUGE\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mkey\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mmygauge\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mvalue\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[34m100\u001b[39;49;00m}, \u001b[37m# a gauge which will be set to given value\u001b[39;49;00m\r\n", + " {\u001b[33m\"\u001b[39;49;00m\u001b[33mtype\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mTIMER\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mkey\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[33m\"\u001b[39;49;00m\u001b[33mmytimer\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[33m\"\u001b[39;49;00m\u001b[33mvalue\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[34m20.2\u001b[39;49;00m}, \u001b[37m# a timer which will add sum and count metrics - assumed millisecs\u001b[39;49;00m\r\n", + " ]\r\n", + " \r\n" + ] + } + ], + "source": [ + "!pygmentize ModelWithMetrics.py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# REST" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---> Installing application source...\n", + "Build completed successfully\n" + ] + } + ], + "source": [ + "!s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT model-with-metrics-rest:0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "672480d40f662cc3d37d999264ea3ecca234eb9bb0cbd685772f1f5a8e515240\r\n" + ] + } + ], + "source": [ + "!docker run --name \"model-with-metrics\" -d --rm -p 5000:5000 model-with-metrics-rest:0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "rm -f proto/prediction*.py\r\n", + "rm -f proto/prediction.proto\r\n", + "rm -rf proto/__pycache__\r\n", + "rm -f fbs/*.py\r\n", + "rm -rf fbs/__pycache__\r\n", + "cp ../../proto/prediction.proto ./proto\r\n", + "python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto\r\n" + ] + } + ], + "source": [ + "!cd ../../../wrappers/testing && make build_protos" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test predict" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "----------------------------------------\r\n", + "SENDING NEW REQUEST:\r\n", + "{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[6.668, 4.529, 4.791, 0.649]]}}\r\n", + "RECEIVED RESPONSE:\r\n", + "{'data': {'names': ['t:0', 't:1', 't:2', 't:3'], 'ndarray': [[6.668, 4.529, 4.791, 0.649]]}, 'meta': {'metrics': [{'key': 'mycounter', 'type': 'COUNTER', 'value': 1}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}}\r\n", + "\r\n", + "Time 0.00577998161315918\r\n" + ] + } + ], + "source": [ + "!python ../../../wrappers/testing/tester.py contract.json 0.0.0.0 5000 -p" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "model-with-metrics\r\n" + ] + } + ], + "source": [ + "!docker rm model-with-metrics --force" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# gRPC" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---> Installing application source...\n", + "Build completed successfully\n" + ] + } + ], + "source": [ + "!s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT model-with-metrics-grpc:0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "b83028c73787596d2a641c84eedc2173951ad6746d78897e459173ce4799a521\r\n" + ] + } + ], + "source": [ + "!docker run --name \"model-with-metrics\" -d --rm -p 5000:5000 model-with-metrics-grpc:0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "rm -f proto/prediction*.py\r\n", + "rm -f proto/prediction.proto\r\n", + "rm -rf proto/__pycache__\r\n", + "rm -f fbs/*.py\r\n", + "rm -rf fbs/__pycache__\r\n", + "cp ../../proto/prediction.proto ./proto\r\n", + "python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto\r\n" + ] + } + ], + "source": [ + "!cd ../../../wrappers/testing && make build_protos" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test predict" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "----------------------------------------\r\n", + "SENDING NEW REQUEST:\r\n", + "data {\r\n", + " names: \"sepal_length\"\r\n", + " names: \"sepal_width\"\r\n", + " names: \"petal_length\"\r\n", + " names: \"petal_width\"\r\n", + " ndarray {\r\n", + " values {\r\n", + " list_value {\r\n", + " values {\r\n", + " number_value: 5.681\r\n", + " }\r\n", + " values {\r\n", + " number_value: 4.435\r\n", + " }\r\n", + " values {\r\n", + " number_value: 6.005\r\n", + " }\r\n", + " values {\r\n", + " number_value: 0.73\r\n", + " }\r\n", + " }\r\n", + " }\r\n", + " }\r\n", + "}\r\n", + "\r\n", + "RECEIVED RESPONSE:\r\n", + "meta {\r\n", + " metrics {\r\n", + " key: \"mycounter\"\r\n", + " value: 1.0\r\n", + " }\r\n", + " metrics {\r\n", + " key: \"mygauge\"\r\n", + " type: GAUGE\r\n", + " value: 100.0\r\n", + " }\r\n", + " metrics {\r\n", + " key: \"mytimer\"\r\n", + " type: TIMER\r\n", + " value: 20.200000762939453\r\n", + " }\r\n", + "}\r\n", + "data {\r\n", + " names: \"t:0\"\r\n", + " names: \"t:1\"\r\n", + " names: \"t:2\"\r\n", + " names: \"t:3\"\r\n", + " ndarray {\r\n", + " values {\r\n", + " list_value {\r\n", + " values {\r\n", + " number_value: 5.681\r\n", + " }\r\n", + " values {\r\n", + " number_value: 4.435\r\n", + " }\r\n", + " values {\r\n", + " number_value: 6.005\r\n", + " }\r\n", + " values {\r\n", + " number_value: 0.73\r\n", + " }\r\n", + " }\r\n", + " }\r\n", + " }\r\n", + "}\r\n", + "\r\n", + "\r\n" + ] + } + ], + "source": [ + "!python ../../../wrappers/testing/tester.py contract.json 0.0.0.0 5000 -p --grpc" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "model-with-metrics\r\n" + ] + } + ], + "source": [ + "!docker rm model-with-metrics --force" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Test using Minikube\n", + "\n", + "**Due to a [minikube/s2i issue](https://github.com/SeldonIO/seldon-core/issues/253) you will need Minikube version 0.25.2**" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "There is a newer version of minikube available (v0.30.0). Download it here:\n", + "https://github.com/kubernetes/minikube/releases/tag/v0.30.0\n", + "\n", + "To disable this notification, run the following:\n", + "minikube config set WantUpdateNotification false\n", + "Starting local Kubernetes v1.9.4 cluster...\n", + "Starting VM...\n", + "Getting VM IP address...\n", + "Moving files into cluster...\n", + "Setting up certs...\n", + "Connecting to cluster...\n", + "Setting up kubeconfig...\n", + "Starting cluster components...\n", + "Kubectl is now configured to use the cluster.\n", + "Loading cached images from config file.\n" + ] + } + ], + "source": [ + "!minikube start --vm-driver kvm2 --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created\r\n" + ] + } + ], + "source": [ + "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "$HELM_HOME has been configured at /home/clive/.helm.\n", + "\n", + "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n", + "\n", + "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n", + "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n", + "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n", + "Happy Helming!\n" + ] + } + ], + "source": [ + "!helm init" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Waiting for deployment \"tiller-deploy\" rollout to finish: 0 of 1 updated replicas are available...\n", + "deployment \"tiller-deploy\" successfully rolled out\n" + ] + } + ], + "source": [ + "!kubectl rollout status deploy/tiller-deploy -n kube-system" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NAME: seldon-core-crd\n", + "LAST DEPLOYED: Sat Nov 3 08:25:19 2018\n", + "NAMESPACE: default\n", + "STATUS: DEPLOYED\n", + "\n", + "RESOURCES:\n", + "==> v1/ServiceAccount\n", + "NAME SECRETS AGE\n", + "seldon-spartakus-volunteer 1 0s\n", + "\n", + "==> v1beta1/ClusterRole\n", + "NAME AGE\n", + "seldon-spartakus-volunteer 0s\n", + "\n", + "==> v1beta1/ClusterRoleBinding\n", + "NAME AGE\n", + "seldon-spartakus-volunteer 0s\n", + "\n", + "==> v1/ConfigMap\n", + "NAME DATA AGE\n", + "seldon-spartakus-config 3 1s\n", + "\n", + "==> v1beta1/CustomResourceDefinition\n", + "NAME AGE\n", + "seldondeployments.machinelearning.seldon.io 0s\n", + "\n", + "==> v1beta1/Deployment\n", + "NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\n", + "seldon-spartakus-volunteer 1 0 0 0 0s\n", + "\n", + "\n", + "NOTES:\n", + "NOTES: TODO\n", + "\n", + "\n", + "NAME: seldon-core\n", + "LAST DEPLOYED: Sat Nov 3 08:25:20 2018\n", + "NAMESPACE: default\n", + "STATUS: DEPLOYED\n", + "\n", + "RESOURCES:\n", + "==> v1/ClusterRoleBinding\n", + "NAME AGE\n", + "seldon-default 0s\n", + "\n", + "==> v1beta1/Role\n", + "NAME AGE\n", + "seldon-local 0s\n", + "\n", + "==> v1/RoleBinding\n", + "NAME AGE\n", + "seldon 0s\n", + "\n", + "==> v1/Service\n", + "NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n", + "seldon-core-seldon-apiserver NodePort 10.108.30.143 8080:30088/TCP,5000:31980/TCP 0s\n", + "seldon-core-redis ClusterIP 10.109.3.187 6379/TCP 0s\n", + "\n", + "==> v1beta1/Deployment\n", + "NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\n", + "seldon-core-seldon-apiserver 1 1 1 0 0s\n", + "seldon-core-seldon-cluster-manager 1 1 1 0 0s\n", + "seldon-core-redis 1 1 1 0 0s\n", + "\n", + "==> v1/Pod(related)\n", + "NAME READY STATUS RESTARTS AGE\n", + "seldon-core-seldon-apiserver-5976c46469-tq2l8 0/1 ContainerCreating 0 0s\n", + "seldon-core-seldon-cluster-manager-55db6c898c-p6mk8 0/1 ContainerCreating 0 0s\n", + "seldon-core-redis-98c4948f7-gm5sv 0/1 ContainerCreating 0 0s\n", + "\n", + "==> v1/ServiceAccount\n", + "NAME SECRETS AGE\n", + "seldon 1 0s\n", + "\n", + "==> v1beta1/ClusterRole\n", + "NAME AGE\n", + "seldon-crd-default 0s\n", + "\n", + "\n", + "NOTES:\n", + "Thank you for installing Seldon Core.\n", + "\n", + "Documentation can be found at https://github.com/SeldonIO/seldon-core\n", + "\n", + "\n", + "\n", + "\n" + ] + } + ], + "source": [ + "!helm install ../../../helm-charts/seldon-core-crd --name seldon-core-crd --set usage_metrics.enabled=true\n", + "!helm install ../../../helm-charts/seldon-core --name seldon-core " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NAME: seldon-core-analytics\n", + "LAST DEPLOYED: Thu Nov 8 13:40:09 2018\n", + "NAMESPACE: seldon\n", + "STATUS: DEPLOYED\n", + "\n", + "RESOURCES:\n", + "==> v1/Secret\n", + "NAME TYPE DATA AGE\n", + "grafana-prom-secret Opaque 1 0s\n", + "\n", + "==> v1/ConfigMap\n", + "NAME DATA AGE\n", + "alertmanager-server-conf 1 0s\n", + "grafana-import-dashboards 7 0s\n", + "prometheus-rules 4 0s\n", + "prometheus-server-conf 1 0s\n", + "\n", + "==> v1/ServiceAccount\n", + "NAME SECRETS AGE\n", + "prometheus 1 0s\n", + "\n", + "==> v1beta1/ClusterRole\n", + "NAME AGE\n", + "prometheus 0s\n", + "\n", + "==> v1beta1/ClusterRoleBinding\n", + "NAME AGE\n", + "prometheus 0s\n", + "\n", + "==> v1/Job\n", + "NAME DESIRED SUCCESSFUL AGE\n", + "grafana-prom-import-dashboards 1 0 0s\n", + "\n", + "==> v1beta1/Deployment\n", + "NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\n", + "alertmanager-deployment 1 1 1 0 0s\n", + "grafana-prom-deployment 1 1 1 0 0s\n", + "prometheus-deployment 1 1 1 0 0s\n", + "\n", + "==> v1beta1/DaemonSet\n", + "NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE\n", + "prometheus-node-exporter 1 1 0 1 0 0s\n", + "\n", + "==> v1/Service\n", + "NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n", + "alertmanager ClusterIP 10.102.58.24 80/TCP 0s\n", + "grafana-prom NodePort 10.100.218.244 80:30688/TCP 0s\n", + "prometheus-node-exporter ClusterIP None 9100/TCP 0s\n", + "prometheus-seldon ClusterIP 10.110.100.130 80/TCP 0s\n", + "\n", + "==> v1/Pod(related)\n", + "NAME READY STATUS RESTARTS AGE\n", + "grafana-prom-import-dashboards-t52ts 0/1 ContainerCreating 0 0s\n", + "alertmanager-deployment-7fbfdfdfb6-4q7hz 0/1 ContainerCreating 0 0s\n", + "grafana-prom-deployment-7b45fb85d4-7znsq 0/1 ContainerCreating 0 0s\n", + "prometheus-node-exporter-p4bwc 0/1 ContainerCreating 0 0s\n", + "prometheus-deployment-cbfd78cc7-nnjd6 0/1 ContainerCreating 0 0s\n", + "\n", + "\n", + "NOTES:\n", + "NOTES: TODO\n", + "\n", + "\n" + ] + } + ], + "source": [ + "!helm install seldon-core-analytics --name seldon-core-analytics --set grafana_prom_admin_password=password --set persistence.enabled=false --repo https://storage.googleapis.com/seldon-charts " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + " * Port forward the dashboard when running\n", + " ```\n", + " kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3000:3000\n", + " ```\n", + " * Visit http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1 and login using \"admin\" and the password you set above when launching with helm." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# REST" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "I1108 13:41:46.852606 1559 build.go:50] Running S2I version \"v1.1.12\"\n", + "I1108 13:41:46.852742 1559 util.go:58] Getting docker credentials for seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\n", + "I1108 13:41:46.852762 1559 util.go:74] Using credentials for pulling seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\n", + "I1108 13:41:46.882915 1559 docker.go:487] Using locally available image \"seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\"\n", + "I1108 13:41:46.884452 1559 build.go:163] \n", + "Builder Image:\t\t\tseldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\n", + "Source:\t\t\t\t.\n", + "Output Image Tag:\t\tmodel-with-metrics-rest:0.1\n", + "Environment:\t\t\tMODEL_NAME=ModelWithMetrics,API_TYPE=REST,SERVICE_TYPE=MODEL,PERSISTENCE=0\n", + "Environment File:\t\tenvironment_rest\n", + "Labels:\t\t\t\t\n", + "Incremental Build:\t\tdisabled\n", + "Remove Old Build:\t\tdisabled\n", + "Builder Pull Policy:\t\tif-not-present\n", + "Previous Image Pull Policy:\tif-not-present\n", + "Quiet:\t\t\t\tdisabled\n", + "Layered Build:\t\t\tdisabled\n", + "Docker Endpoint:\t\ttcp://192.168.39.50:2376\n", + "Docker Pull Config:\t\t/home/clive/.docker/config.json\n", + "Docker Pull User:\t\tcliveseldon\n", + "\n", + "I1108 13:41:46.885812 1559 docker.go:487] Using locally available image \"seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\"\n", + "I1108 13:41:46.889703 1559 docker.go:487] Using locally available image \"seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\"\n", + "I1108 13:41:46.889713 1559 docker.go:718] Image sha256:29d7be19772bfb306b12e4ece8ba007330c72a5c2c574ba3769ba69ad06ea898 contains io.openshift.s2i.scripts-url set to \"image:///s2i/bin\"\n", + "I1108 13:41:46.889939 1559 scm.go:20] DownloadForSource .\n", + "I1108 13:41:46.889999 1559 sti.go:204] Preparing to build model-with-metrics-rest:0.1\n", + "I1108 13:41:46.890159 1559 download.go:38] Copying sources from \".\" to \"/tmp/s2i141179583/upload/src\"\n", + "I1108 13:41:46.890265 1559 fs.go:260] F \"contract.json\" -> \"/tmp/s2i141179583/upload/src/contract.json\"\n", + "I1108 13:41:46.890325 1559 fs.go:260] F \"deployment-rest.json\" -> \"/tmp/s2i141179583/upload/src/deployment-rest.json\"\n", + "I1108 13:41:46.890358 1559 fs.go:260] F \"modelWithMetrics.ipynb\" -> \"/tmp/s2i141179583/upload/src/modelWithMetrics.ipynb\"\n", + "I1108 13:41:46.890427 1559 fs.go:260] F \"environment_rest\" -> \"/tmp/s2i141179583/upload/src/environment_rest\"\n", + "I1108 13:41:46.890459 1559 fs.go:260] F \"environment_rest~\" -> \"/tmp/s2i141179583/upload/src/environment_rest~\"\n", + "I1108 13:41:46.890492 1559 fs.go:260] F \"ModelWithMetrics.py~\" -> \"/tmp/s2i141179583/upload/src/ModelWithMetrics.py~\"\n", + "I1108 13:41:46.890525 1559 fs.go:260] F \"deployment-grpc.json\" -> \"/tmp/s2i141179583/upload/src/deployment-grpc.json\"\n", + "I1108 13:41:46.891110 1559 fs.go:260] F \"deployment-rest.json~\" -> \"/tmp/s2i141179583/upload/src/deployment-rest.json~\"\n", + "I1108 13:41:46.891147 1559 fs.go:247] D \".ipynb_checkpoints\" -> \"/tmp/s2i141179583/upload/src/.ipynb_checkpoints\"\n", + "I1108 13:41:46.891200 1559 fs.go:260] F \".ipynb_checkpoints/modelWithMetrics-checkpoint.ipynb\" -> \"/tmp/s2i141179583/upload/src/.ipynb_checkpoints/modelWithMetrics-checkpoint.ipynb\"\n", + "I1108 13:41:46.891271 1559 fs.go:260] F \"environment_grpc~\" -> \"/tmp/s2i141179583/upload/src/environment_grpc~\"\n", + "I1108 13:41:46.891303 1559 fs.go:260] F \"ModelWithMetrics.py\" -> \"/tmp/s2i141179583/upload/src/ModelWithMetrics.py\"\n", + "I1108 13:41:46.891334 1559 fs.go:260] F \"environment_grpc\" -> \"/tmp/s2i141179583/upload/src/environment_grpc\"\n", + "I1108 13:41:46.891381 1559 fs.go:260] F \"deployment-grpc.json~\" -> \"/tmp/s2i141179583/upload/src/deployment-grpc.json~\"\n", + "I1108 13:41:46.891416 1559 install.go:261] Using \"assemble\" installed from \"image:///s2i/bin/assemble\"\n", + "I1108 13:41:46.891446 1559 install.go:261] Using \"run\" installed from \"image:///s2i/bin/run\"\n", + "I1108 13:41:46.891457 1559 install.go:261] Using \"save-artifacts\" installed from \"image:///s2i/bin/save-artifacts\"\n", + "I1108 13:41:46.891504 1559 ignore.go:64] .s2iignore file does not exist\n", + "I1108 13:41:46.891510 1559 sti.go:213] Clean build will be performed\n", + "I1108 13:41:46.891515 1559 sti.go:216] Performing source build from .\n", + "I1108 13:41:46.891532 1559 sti.go:227] Running \"assemble\" in \"model-with-metrics-rest:0.1\"\n", + "I1108 13:41:46.891536 1559 sti.go:573] Using image name seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\n", + "I1108 13:41:46.892846 1559 docker.go:487] Using locally available image \"seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT\"\n", + "I1108 13:41:46.892879 1559 sti.go:453] No user environment provided (no environment file found in application sources)\n", + "I1108 13:41:46.892909 1559 sti.go:691] starting the source uploading ...\n", + "I1108 13:41:46.892920 1559 tar.go:217] Adding \"/tmp/s2i141179583/upload\" to tar ...\n", + "I1108 13:41:46.892967 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/scripts as scripts\n", + "I1108 13:41:46.895541 1559 docker.go:718] Image sha256:29d7be19772bfb306b12e4ece8ba007330c72a5c2c574ba3769ba69ad06ea898 contains io.openshift.s2i.scripts-url set to \"image:///s2i/bin\"\n", + "I1108 13:41:46.895550 1559 docker.go:793] Base directory for S2I scripts is '/s2i/bin'. Untarring destination is '/tmp'.\n", + "I1108 13:41:46.895558 1559 docker.go:949] Setting \"/bin/sh -c tar -C /tmp -xf - && /s2i/bin/assemble\" command for container ...\n", + "I1108 13:41:46.895631 1559 docker.go:958] Creating container with options {Name:\"s2i_seldonio_seldon_core_s2i_python3_0_3_SNAPSHOT_beeacde3\" Config:{Hostname: Domainname: User: AttachStdin:false AttachStdout:true AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:true StdinOnce:true Env:[MODEL_NAME=ModelWithMetrics API_TYPE=REST SERVICE_TYPE=MODEL PERSISTENCE=0] Cmd:[/bin/sh -c tar -C /tmp -xf - && /s2i/bin/assemble] Healthcheck: ArgsEscaped:false Image:seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[] StopSignal: StopTimeout: Shell:[]} HostConfig:&{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:67108864 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DiskQuota:0 KernelMemory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness: OomKillDisable: PidsLimit:0 Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] Init:}} ...\n", + "I1108 13:41:46.929308 1559 docker.go:990] Attaching to container \"d661ca52e1d7662ef0badcb2be3688f90759c9756498c0aee3a0089e2a99190b\" ...\n", + "I1108 13:41:46.944928 1559 docker.go:1001] Starting container \"d661ca52e1d7662ef0badcb2be3688f90759c9756498c0aee3a0089e2a99190b\" ...\n", + "I1108 13:41:47.121961 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src as src\n", + "I1108 13:41:47.122034 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/.ipynb_checkpoints as src/.ipynb_checkpoints\n", + "I1108 13:41:47.122107 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/.ipynb_checkpoints/modelWithMetrics-checkpoint.ipynb as src/.ipynb_checkpoints/modelWithMetrics-checkpoint.ipynb\n", + "I1108 13:41:47.122217 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/ModelWithMetrics.py as src/ModelWithMetrics.py\n", + "I1108 13:41:47.122267 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/ModelWithMetrics.py~ as src/ModelWithMetrics.py~\n", + "I1108 13:41:47.122311 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/contract.json as src/contract.json\n", + "I1108 13:41:47.122355 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/deployment-grpc.json as src/deployment-grpc.json\n", + "I1108 13:41:47.122395 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/deployment-grpc.json~ as src/deployment-grpc.json~\n", + "I1108 13:41:47.122506 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/deployment-rest.json as src/deployment-rest.json\n", + "I1108 13:41:47.122627 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/deployment-rest.json~ as src/deployment-rest.json~\n", + "I1108 13:41:47.122690 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/environment_grpc as src/environment_grpc\n", + "I1108 13:41:47.122890 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/environment_grpc~ as src/environment_grpc~\n", + "I1108 13:41:47.122940 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/environment_rest as src/environment_rest\n", + "I1108 13:41:47.123039 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/environment_rest~ as src/environment_rest~\n", + "I1108 13:41:47.123120 1559 tar.go:312] Adding to tar: /tmp/s2i141179583/upload/src/modelWithMetrics.ipynb as src/modelWithMetrics.ipynb\n", + "I1108 13:41:47.129342 1559 sti.go:699] ---> Installing application source...\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "I1108 13:41:47.157389 1559 docker.go:1032] Waiting for container \"d661ca52e1d7662ef0badcb2be3688f90759c9756498c0aee3a0089e2a99190b\" to stop ...\n", + "I1108 13:41:47.244738 1559 docker.go:1057] Invoking PostExecute function\n", + "I1108 13:41:47.244769 1559 postexecutorstep.go:68] Skipping step: store previous image\n", + "I1108 13:41:47.244773 1559 postexecutorstep.go:117] Executing step: commit image\n", + "I1108 13:41:47.246352 1559 postexecutorstep.go:522] Checking for new Labels to apply... \n", + "I1108 13:41:47.246362 1559 postexecutorstep.go:530] Creating the download path '/tmp/s2i141179583/metadata'\n", + "I1108 13:41:47.246431 1559 postexecutorstep.go:464] Downloading file \"/tmp/.s2i/image_metadata.json\"\n", + "I1108 13:41:47.284759 1559 postexecutorstep.go:538] unable to download and extract 'image_metadata.json' ... continuing\n", + "I1108 13:41:47.287415 1559 docker.go:1091] Committing container with dockerOpts: {Reference:model-with-metrics-rest:0.1 Comment: Author: Changes:[] Pause:false Config:0xc420092c80}, config: {Hostname: Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[MODEL_NAME=ModelWithMetrics API_TYPE=REST SERVICE_TYPE=MODEL PERSISTENCE=0] Cmd:[/s2i/bin/run] Healthcheck: ArgsEscaped:false Image: Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[io.openshift.s2i.scripts-url:image:///s2i/bin io.openshift.s2i.build.source-location:. io.k8s.display-name:model-with-metrics-rest:0.1 io.openshift.s2i.build.image:seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT] StopSignal: StopTimeout: Shell:[]}\n", + "I1108 13:41:47.356945 1559 postexecutorstep.go:392] Executing step: report success\n", + "I1108 13:41:47.356968 1559 postexecutorstep.go:397] Successfully built model-with-metrics-rest:0.1\n", + "I1108 13:41:47.356978 1559 postexecutorstep.go:93] Skipping step: remove previous image\n", + "I1108 13:41:47.357049 1559 docker.go:968] Removing container \"d661ca52e1d7662ef0badcb2be3688f90759c9756498c0aee3a0089e2a99190b\" ...\n", + "I1108 13:41:47.380567 1559 docker.go:978] Removed container \"d661ca52e1d7662ef0badcb2be3688f90759c9756498c0aee3a0089e2a99190b\"\n", + "I1108 13:41:47.380704 1559 cleanup.go:33] Removing temporary directory /tmp/s2i141179583\n", + "I1108 13:41:47.380723 1559 fs.go:302] Removing directory '/tmp/s2i141179583'\n", + "I1108 13:41:47.381363 1559 build.go:175] Build completed successfully\n" + ] + } + ], + "source": [ + "!eval $(minikube docker-env) && s2i build -E environment_rest . seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT model-with-metrics-rest:0.1 --loglevel 5" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "seldondeployment.machinelearning.seldon.io/mymodel created\r\n" + ] + } + ], + "source": [ + "!kubectl create -f deployment-rest.json" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Wait until ready (replicas == replicasAvailable)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "map[predictorStatus:[map[name:mymodel-mymodel-svc-orch replicas:1 replicasAvailable:1] map[replicasAvailable:1 name:mymodel-mymodel-complex-model-0 replicas:1]] state:Available]" + ] + } + ], + "source": [ + "!kubectl get seldondeployments mymodel -o jsonpath='{.status}' " + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "rm -f proto/prediction*.py\r\n", + "rm -f proto/prediction.proto\r\n", + "rm -rf proto/__pycache__\r\n", + "mkdir -p ./proto\r\n", + "touch ./proto/__init__.py\r\n", + "cp ../../proto/prediction.proto ./proto\r\n", + "python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto\r\n" + ] + } + ], + "source": [ + "!cd ../../../util/api_tester && make build_protos " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test predict" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "----------------------------------------\n", + "SENDING NEW REQUEST:\n", + "{'meta': {}, 'data': {'names': ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], 'ndarray': [[7.974098303518131, 3.149273146516985, 8.219107590512102, 2.3086360840551796]]}}\n", + "Getting token from http://192.168.39.50:31307/oauth/token\n", + "{\"access_token\":\"343409a6-0abd-4b8c-ad83-dd1f9657fef4\",\"token_type\":\"bearer\",\"expires_in\":42651,\"scope\":\"read write\"}\n", + "RECEIVED RESPONSE:\n", + "{'meta': {'puid': 'l05rsshl95btcv6rvr2k6ig3oa', 'tags': {}, 'routing': {}, 'requestPath': {'complex-model': 'model-with-metrics-rest:0.1'}, 'metrics': [{'key': 'mycounter', 'type': 'COUNTER', 'value': 1.0}, {'key': 'mygauge', 'type': 'GAUGE', 'value': 100.0}, {'key': 'mytimer', 'type': 'TIMER', 'value': 20.2}]}, 'data': {'names': ['t:0', 't:1', 't:2', 't:3'], 'ndarray': [[7.974098303518131, 3.149273146516985, 8.219107590512102, 2.3086360840551796]]}}\n", + "\n" + ] + } + ], + "source": [ + "!python ../../../util/api_tester/api-tester.py contract.json \\\n", + " `minikube ip` `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[0].nodePort}'` \\\n", + " --oauth-key oauth-key --oauth-secret oauth-secret -p" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "seldondeployment.machinelearning.seldon.io \"mymodel\" deleted\r\n" + ] + } + ], + "source": [ + "!kubectl delete -f deployment-rest.json" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# gRPC" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---> Installing application source...\n", + "Build completed successfully\n" + ] + } + ], + "source": [ + "!eval $(minikube docker-env) && s2i build -E environment_grpc . seldonio/seldon-core-s2i-python3:0.3-SNAPSHOT model-with-metrics-grpc:0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "seldondeployment.machinelearning.seldon.io/mymodel created\r\n" + ] + } + ], + "source": [ + "!kubectl create -f deployment-grpc.json" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Wait until ready (replicas == replicasAvailable)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "map[predictorStatus:[map[replicasAvailable:1 name:mymodel-mymodel-svc-orch replicas:1] map[name:mymodel-mymodel-complex-model-0 replicas:1 replicasAvailable:1]] state:Available]" + ] + } + ], + "source": [ + "!kubectl get seldondeployments mymodel -o jsonpath='{.status}' " + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "rm -f proto/prediction*.py\r\n", + "rm -f proto/prediction.proto\r\n", + "rm -rf proto/__pycache__\r\n", + "mkdir -p ./proto\r\n", + "touch ./proto/__init__.py\r\n", + "cp ../../proto/prediction.proto ./proto\r\n", + "python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto\r\n" + ] + } + ], + "source": [ + "!cd ../../../util/api_tester && make build_protos " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Validate on Grafana\n", + "\n", + "To check the metrics have appeared on Prometheus and are available in Grafana you could create a new graph in a dashboard and use the query:\n", + "\n", + "```\n", + "mycounter_total\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test predict" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "----------------------------------------\n", + "SENDING NEW REQUEST:\n", + "data {\n", + " names: \"sepal_length\"\n", + " names: \"sepal_width\"\n", + " names: \"petal_length\"\n", + " names: \"petal_width\"\n", + " ndarray {\n", + " values {\n", + " list_value {\n", + " values {\n", + " number_value: 5.018607555163812\n", + " }\n", + " values {\n", + " number_value: 4.150787936306038\n", + " }\n", + " values {\n", + " number_value: 9.145665131982822\n", + " }\n", + " values {\n", + " number_value: 0.12728175889218607\n", + " }\n", + " }\n", + " }\n", + " }\n", + "}\n", + "\n", + "Getting token from http://192.168.39.50:31307/oauth/token\n", + "{\"access_token\":\"343409a6-0abd-4b8c-ad83-dd1f9657fef4\",\"token_type\":\"bearer\",\"expires_in\":43122,\"scope\":\"read write\"}\n", + "RECEIVED RESPONSE:\n", + "meta {\n", + " puid: \"6k4ba0l998145lnokbc4t2ncff\"\n", + " requestPath {\n", + " key: \"complex-model\"\n", + " value: \"model-with-metrics-grpc:0.1\"\n", + " }\n", + " metrics {\n", + " key: \"mycounter\"\n", + " value: 1.0\n", + " }\n", + " metrics {\n", + " key: \"mygauge\"\n", + " type: GAUGE\n", + " value: 100.0\n", + " }\n", + " metrics {\n", + " key: \"mytimer\"\n", + " type: TIMER\n", + " value: 20.200000762939453\n", + " }\n", + "}\n", + "data {\n", + " names: \"t:0\"\n", + " names: \"t:1\"\n", + " names: \"t:2\"\n", + " names: \"t:3\"\n", + " ndarray {\n", + " values {\n", + " list_value {\n", + " values {\n", + " number_value: 5.018607555163812\n", + " }\n", + " values {\n", + " number_value: 4.150787936306038\n", + " }\n", + " values {\n", + " number_value: 9.145665131982822\n", + " }\n", + " values {\n", + " number_value: 0.12728175889218607\n", + " }\n", + " }\n", + " }\n", + " }\n", + "}\n", + "\n", + "\n" + ] + } + ], + "source": [ + "!python ../../../util/api_tester/api-tester.py contract.json \\\n", + " `minikube ip` `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[1].nodePort}'` \\\n", + " --oauth-key oauth-key --oauth-secret oauth-secret -p --grpc --oauth-port `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[0].nodePort}'`" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "seldondeployment.machinelearning.seldon.io \"mymodel\" deleted\r\n" + ] + } + ], + "source": [ + "!kubectl delete -f deployment-grpc.json" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleting local Kubernetes cluster...\n", + "Machine deleted.\n" + ] + } + ], + "source": [ + "!minikube delete" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/openapi/apife.oas3.json b/openapi/apife.oas3.json index e087da760f..ca7ecee00f 100644 --- a/openapi/apife.oas3.json +++ b/openapi/apife.oas3.json @@ -121,6 +121,30 @@ "AnyValue": { "description": "Can be anything: string, number, array, object, etc." }, + "MetricType": { + "type": "string", + "enum": [ + "COUNTER", + "GAUGE", + "TIMER" + ], + "default": "COUNTER" + }, + "Metric": { + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/MetricType" + }, + "key": { + "type": "string" + }, + "value": { + "type": "number", + "format": "float" + } + } + }, "DefaultData": { "type": "object", "properties": { @@ -192,6 +216,12 @@ "example": { "classifier": "seldonio/mock_classifier:1.0" } + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Metric" + } } } }, diff --git a/openapi/components.json b/openapi/components.json index b5a06acf1b..2109f3cd6d 100644 --- a/openapi/components.json +++ b/openapi/components.json @@ -12,6 +12,30 @@ "AnyValue": { "description": "Can be anything: string, number, array, object, etc." }, + "MetricType": { + "type": "string", + "enum": [ + "COUNTER", + "GAUGE", + "TIMER" + ], + "default": "COUNTER" + }, + "Metric": { + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/MetricType" + }, + "key": { + "type": "string" + }, + "value": { + "type": "number", + "format": "float" + } + } + }, "DefaultData": { "type": "object", "properties": { @@ -83,6 +107,12 @@ "example": { "classifier": "seldonio/mock_classifier:1.0" } + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Metric" + } } } }, diff --git a/openapi/engine.oas3.json b/openapi/engine.oas3.json index f8f761e89d..bfd0326951 100644 --- a/openapi/engine.oas3.json +++ b/openapi/engine.oas3.json @@ -141,6 +141,30 @@ "AnyValue": { "description": "Can be anything: string, number, array, object, etc." }, + "MetricType": { + "type": "string", + "enum": [ + "COUNTER", + "GAUGE", + "TIMER" + ], + "default": "COUNTER" + }, + "Metric": { + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/MetricType" + }, + "key": { + "type": "string" + }, + "value": { + "type": "number", + "format": "float" + } + } + }, "DefaultData": { "type": "object", "properties": { @@ -212,6 +236,12 @@ "example": { "classifier": "seldonio/mock_classifier:1.0" } + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Metric" + } } } }, diff --git a/openapi/wrapper.oas3.json b/openapi/wrapper.oas3.json index 3aeff429c6..1d01dc70c7 100644 --- a/openapi/wrapper.oas3.json +++ b/openapi/wrapper.oas3.json @@ -456,6 +456,30 @@ "AnyValue": { "description": "Can be anything: string, number, array, object, etc." }, + "MetricType": { + "type": "string", + "enum": [ + "COUNTER", + "GAUGE", + "TIMER" + ], + "default": "COUNTER" + }, + "Metric": { + "type": "object", + "properties": { + "type": { + "$ref": "#/components/schemas/MetricType" + }, + "key": { + "type": "string" + }, + "value": { + "type": "number", + "format": "float" + } + } + }, "DefaultData": { "type": "object", "properties": { @@ -527,6 +551,12 @@ "example": { "classifier": "seldonio/mock_classifier:1.0" } + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Metric" + } } } }, diff --git a/proto/prediction.proto b/proto/prediction.proto index c2360bbd85..a504509558 100644 --- a/proto/prediction.proto +++ b/proto/prediction.proto @@ -38,6 +38,18 @@ message Meta { map tags = 2; map routing = 3; map requestPath = 4; + repeated Metric metrics = 5; +} + +message Metric { + enum MetricType { + COUNTER = 0; + GAUGE = 1; + TIMER = 2; + } + string key = 1; + MetricType type = 2; + float value = 3; } message SeldonMessageList { diff --git a/wrappers/python/Makefile b/wrappers/python/Makefile index 53cf2a150b..456865e687 100644 --- a/wrappers/python/Makefile +++ b/wrappers/python/Makefile @@ -8,10 +8,19 @@ build_proto: cp ../../proto/prediction.proto proto python -m grpc.tools.protoc -I./ --python_out=./ --grpc_python_out=./ ./proto/prediction.proto +build_fbs: + cp ../../fbs/prediction.fbs fbs + flatc --python -o fbs fbs/prediction.fbs + seldon.json: cp ../../openapi/wrapper.oas3.json seldon.json +test: + PYTHONPATH=. pytest + clean: rm -f proto/prediction.proto rm -f proto/prediction_pb2_grpc.py rm -f proto/prediction_pb2.py + rm -f fbs/*.py* + rm -f fbs/prediction.fbs diff --git a/wrappers/python/metrics.py b/wrappers/python/metrics.py new file mode 100644 index 0000000000..7df7c1a68c --- /dev/null +++ b/wrappers/python/metrics.py @@ -0,0 +1,45 @@ +from microservice import SeldonMicroserviceException +import json + +COUNTER = "COUNTER" +GAUGE = "GAUGE" +TIMER = "TIMER" + +def create_counter(key,value): + test = value + 1 + return {"key":key,"type":COUNTER,"value":value} + +def create_gauge(key,value): + test = value + 1 + return {"key":key,"type":GAUGE,"value":value} + +def create_timer(key,value): + test = value + 1 + return {"key":key,"type":TIMER,"value":value} + +def validate_metrics(metrics): + if isinstance(metrics, (list,)): + for metric in metrics: + if not ("key" in metric and "value" in metric and "type" in metric): + return False + if not (metric["type"] == COUNTER or metric["type"] == GAUGE or metric["type"] == TIMER): + return False + try: + metric["value"] + 1 + except TypeError: + return False + else: + return False + return True + +def get_custom_metrics(component): + if hasattr(component,"metrics"): + metrics = component.metrics() + if not validate_metrics(metrics): + jStr = json.dumps(metrics) + raise SeldonMicroserviceException("Bad metric created during request: "+jStr,reason="MICROSERVICE_BAD_METRIC") + return metrics + else: + return None + + diff --git a/wrappers/python/microservice.py b/wrappers/python/microservice.py index 17b3e3d660..9de86828be 100644 --- a/wrappers/python/microservice.py +++ b/wrappers/python/microservice.py @@ -39,15 +39,16 @@ def startServers(target1, target2): class SeldonMicroserviceException(Exception): status_code = 400 - def __init__(self, message, status_code= None, payload=None): + def __init__(self, message, status_code= None, payload=None, reason="MICROSERVICE_BAD_DATA"): Exception.__init__(self) self.message = message if status_code is not None: self.status_code = status_code self.payload = payload + self.reason=reason def to_dict(self): - rv = {"status":{"status":1,"info":self.message,"code":-1,"reason":"MICROSERVICE_BAD_DATA"}} + rv = {"status":{"status":1,"info":self.message,"code":-1,"reason":self.reason}} return rv def sanity_check_request(req): @@ -76,6 +77,13 @@ def extract_message(): raise SeldonMicroserviceException("Invalid Data Format") return message +def get_custom_tags(component): + if hasattr(component,"tags"): + return component.tags() + else: + return None + + def array_to_list_value(array,lv=None): if lv is None: lv = ListValue() diff --git a/wrappers/python/model_microservice.py b/wrappers/python/model_microservice.py index 9a38686da5..7340c5331a 100644 --- a/wrappers/python/model_microservice.py +++ b/wrappers/python/model_microservice.py @@ -1,9 +1,11 @@ from proto import prediction_pb2, prediction_pb2_grpc from microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ - SeldonMicroserviceException + SeldonMicroserviceException, get_custom_tags +from metrics import get_custom_metrics import grpc from concurrent import futures +from google.protobuf import json_format from flask import jsonify, Flask, send_from_directory from flask_cors import CORS @@ -78,7 +80,14 @@ def Predict(): data = array_to_rest_datadef(predictions, class_names, datadef) - return jsonify({"data":data}) + response = {"data":data,"meta":{}} + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) @app.route("/send-feedback",methods=["GET","POST"]) def SendFeedback(): @@ -116,7 +125,19 @@ def Predict(self,request,context): class_names = [] data = array_to_grpc_datadef(predictions, class_names, request.data.WhichOneof("data_oneof")) - return prediction_pb2.SeldonMessage(data=data) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson,meta) + + return prediction_pb2.SeldonMessage(data=data,meta=meta) def SendFeedback(self,feedback,context): datadef_request = feedback.request.data diff --git a/wrappers/python/router_microservice.py b/wrappers/python/router_microservice.py index 34a0672f24..480d92dd01 100644 --- a/wrappers/python/router_microservice.py +++ b/wrappers/python/router_microservice.py @@ -1,7 +1,8 @@ from proto import prediction_pb2, prediction_pb2_grpc from microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ - SeldonMicroserviceException + SeldonMicroserviceException, get_custom_tags +from metrics import get_custom_metrics import grpc from concurrent import futures @@ -63,7 +64,14 @@ def Route(): data = array_to_rest_datadef(routing, class_names, datadef) - return jsonify({"data":data}) + response = {"data":data,"meta":{}} + tags = get_custom_tags(user_router) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_router) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) @app.route("/send-feedback",methods=["GET","POST"]) def SendFeedback(): @@ -109,7 +117,19 @@ def Route(self,request,context): class_names = [] data = array_to_grpc_datadef(routing, class_names, request.data.WhichOneof("data_oneof")) - return prediction_pb2.SeldonMessage(data=data) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson,meta) + + return prediction_pb2.SeldonMessage(data=data,meta=meta) def SendFeedback(self,feedback,context): datadef_request = feedback.request.data diff --git a/wrappers/python/test_metrics.py b/wrappers/python/test_metrics.py new file mode 100644 index 0000000000..3cd56f921f --- /dev/null +++ b/wrappers/python/test_metrics.py @@ -0,0 +1,97 @@ +from metrics import * +import pytest +from microservice import SeldonMicroserviceException +from google.protobuf import json_format +from proto import prediction_pb2, prediction_pb2_grpc +import json + +def test_create_counter(): + v = create_counter("k",1) + assert v["type"] == "COUNTER" + +def test_create_counter_invalid_value(): + with pytest.raises(TypeError): + v = create_counter("k","invalid") + +def test_create_timer(): + v = create_timer("k",1) + assert v["type"] == "TIMER" + +def test_create_timer_invalid_value(): + with pytest.raises(TypeError): + v = create_timer("k","invalid") + +def test_create_gauge(): + v = create_gauge("k",1) + assert v["type"] == "GAUGE" + +def test_create_gauge_invalid_value(): + with pytest.raises(TypeError): + v = create_gauge("k","invalid") + + +def test_validate_ok(): + assert validate_metrics([{"type":COUNTER,"key":"a","value":1}]) == True + +def test_validate_bad_type(): + assert validate_metrics([{"type":"ABC","key":"a","value":1}]) == False + +def test_validate_no_type(): + assert validate_metrics([{"key":"a","value":1}]) == False + +def test_validate_no_key(): + assert validate_metrics([{"type":COUNTER,"value":1}]) == False + +def test_validate_no_value(): + assert validate_metrics([{"type":COUNTER,"key":"a"}]) == False + +def test_validate_bad_value(): + assert validate_metrics([{"type":COUNTER,"key":"a","value":"1"}]) == False + +def test_validate_no_list(): + assert validate_metrics({"type":COUNTER,"key":"a","value":1}) == False + + +class Component(object): + + def __init__(self,ok=True): + self.ok = ok + + def metrics(self): + if self.ok: + return [{"type":COUNTER,"key":"a","value":1}] + else: + return [{"type":"bad","key":"a","value":1}] + + +def test_component_ok(): + c = Component(True) + assert get_custom_metrics(c) == c.metrics() + +def test_component_bad(): + with pytest.raises(SeldonMicroserviceException): + c = Component(False) + get_custom_metrics(c) + +def test_proto_metrics(): + metrics = [{"type":"COUNTER","key":"a","value":1}] + meta = prediction_pb2.Meta() + for metric in metrics: + mpb2 = meta.metrics.add() + json_format.ParseDict(metric,mpb2) + + +def test_proto_tags(): + metric = {"tags":{"t1":"t2"},"metrics":[{"type":"COUNTER","key":"mycounter","value":1.2},{"type":"GAUGE","key":"mygauge","value":1.2},{"type":"TIMER","key":"mytimer","value":1.2}]} + meta = prediction_pb2.Meta() + json_format.ParseDict(metric,meta) + jStr = json_format.MessageToJson(meta) + j = json.loads(jStr) + assert "mycounter" == j["metrics"][0]["key"] + assert 1.2 == pytest.approx(j["metrics"][0]["value"],0.01) + assert "GAUGE" == j["metrics"][1]["type"] + assert "mygauge" == j["metrics"][1]["key"] + assert 1.2 == pytest.approx(j["metrics"][1]["value"],0.01) + assert "TIMER" == j["metrics"][2]["type"] + assert "mytimer" == j["metrics"][2]["key"] + assert 1.2 == pytest.approx(j["metrics"][2]["value"],0.01) diff --git a/wrappers/python/test_model_microservice.py b/wrappers/python/test_model_microservice.py new file mode 100644 index 0000000000..29ea3c2cb1 --- /dev/null +++ b/wrappers/python/test_model_microservice.py @@ -0,0 +1,62 @@ +import pytest +from model_microservice import get_rest_microservice +import json + +class UserObject(object): + def __init__(self,metrics_ok=True): + self.metrics_ok = metrics_ok + + def predict(self,X,features_names): + """ + Return a prediction. + + Parameters + ---------- + X : array-like + feature_names : array of feature names (optional) + """ + print("Predict called - will run identity function") + print(X) + return X + + def tags(self): + return {"mytag":1} + + def metrics(self): + if self.metrics_ok: + return [{"type":"COUNTER","key":"mycounter","value":1}] + else: + return [{"type":"BAD","key":"mycounter","value":1}] + + + +def test_model_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/predict?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag":1} + assert j["meta"]["metrics"] == user_object.metrics() + +def test_model_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/predict?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + +def test_model_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/predict?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + diff --git a/wrappers/python/test_router_microservice.py b/wrappers/python/test_router_microservice.py new file mode 100644 index 0000000000..9b3150e2f4 --- /dev/null +++ b/wrappers/python/test_router_microservice.py @@ -0,0 +1,53 @@ +import pytest +from router_microservice import get_rest_microservice +import json + +class UserObject(object): + def __init__(self,metrics_ok=True): + self.metrics_ok = metrics_ok + + def route(self,X,features_names): + return 22 + + def tags(self): + return {"mytag":1} + + def metrics(self): + if self.metrics_ok: + return [{"type":"COUNTER","key":"mycounter","value":1}] + else: + return [{"type":"BAD","key":"mycounter","value":1}] + + + +def test_router_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/route?json={"data":{"ndarray":[2]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag":1} + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["ndarray"] == [[22]] + +def test_router_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/route?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + +def test_router_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/route?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + diff --git a/wrappers/python/test_transformer_microservice.py b/wrappers/python/test_transformer_microservice.py new file mode 100644 index 0000000000..bb4f25d0f7 --- /dev/null +++ b/wrappers/python/test_transformer_microservice.py @@ -0,0 +1,91 @@ +import pytest +from transformer_microservice import get_rest_microservice +import json + +class UserObject(object): + def __init__(self,metrics_ok=True): + self.metrics_ok = metrics_ok + + def transform_input(self,X,features_names): + print("Transform input called - will run identity function") + print(X) + return X + + def transform_output(self,X,features_names): + print("Transform output called - will run identity function") + print(X) + return X + + def tags(self): + return {"mytag":1} + + def metrics(self): + if self.metrics_ok: + return [{"type":"COUNTER","key":"mycounter","value":1}] + else: + return [{"type":"BAD","key":"mycounter","value":1}] + + + +def test_transformer_input_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/transform-input?json={"data":{"ndarray":[1]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag":1} + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["ndarray"] == [1] + +def test_tranform_input_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/transform-input?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + +def test_transform_input_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/transform-input?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + +def test_transformer_output_ok(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/transform-output?json={"data":{"ndarray":[1]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 200 + assert j["meta"]["tags"] == {"mytag":1} + assert j["meta"]["metrics"] == user_object.metrics() + assert j["data"]["ndarray"] == [1] + +def test_tranform_output_no_json(): + user_object = UserObject() + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + uo = UserObject() + rv = client.get('/transform-output?') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + +def test_transform_output_bad_metrics(): + user_object = UserObject(metrics_ok=False) + app = get_rest_microservice(user_object,debug=True) + client = app.test_client() + rv = client.get('/transform-output?json={"data":{"ndarray":[]}}') + j = json.loads(rv.data) + print(j) + assert rv.status_code == 400 + diff --git a/wrappers/python/transformer_microservice.py b/wrappers/python/transformer_microservice.py index 996f95f43d..adfaaf1c20 100644 --- a/wrappers/python/transformer_microservice.py +++ b/wrappers/python/transformer_microservice.py @@ -1,7 +1,8 @@ from proto import prediction_pb2, prediction_pb2_grpc from microservice import extract_message, sanity_check_request, rest_datadef_to_array, \ array_to_rest_datadef, grpc_datadef_to_array, array_to_grpc_datadef, \ - SeldonMicroserviceException + SeldonMicroserviceException, get_custom_tags +from metrics import get_custom_metrics import grpc from concurrent import futures @@ -73,7 +74,14 @@ def TransformInput(): data = array_to_rest_datadef(transformed, new_feature_names, datadef) - return jsonify({"data":data}) + response = {"data":data,"meta":{}} + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) @app.route("/transform-output",methods=["GET","POST"]) def TransformOutput(): @@ -89,7 +97,14 @@ def TransformOutput(): data = array_to_rest_datadef(transformed, new_class_names, datadef) - return jsonify({"data":data}) + response = {"data":data,"meta":{}} + tags = get_custom_tags(user_model) + if tags: + response["meta"]["tags"] = tags + metrics = get_custom_metrics(user_model) + if metrics: + response["meta"]["metrics"] = metrics + return jsonify(response) return app @@ -112,7 +127,19 @@ def TransformInput(self,request,context): feature_names = get_feature_names(self.user_model, datadef.names) data = array_to_grpc_datadef(transformed, feature_names, request.data.WhichOneof("data_oneof")) - return prediction_pb2.SeldonMessage(data=data) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson,meta) + + return prediction_pb2.SeldonMessage(data=data,meta=meta) def TransformOutput(self,request,context): datadef = request.data @@ -123,7 +150,19 @@ def TransformOutput(self,request,context): class_names = get_class_names(self.user_model, datadef.names) data = array_to_grpc_datadef(transformed, class_names, request.data.WhichOneof("data_oneof")) - return prediction_pb2.SeldonMessage(data=data) + + # Construct meta data + meta = prediction_pb2.Meta() + metaJson = {} + tags = get_custom_tags(self.user_model) + if tags: + metaJson["tags"] = tags + metrics = get_custom_metrics(self.user_model) + if metrics: + metaJson["metrics"] = metrics + json_format.ParseDict(metaJson,meta) + + return prediction_pb2.SeldonMessage(data=data,meta=meta) def get_grpc_server(user_model,debug=False,annotations={}): seldon_model = SeldonTransformerGRPC(user_model) diff --git a/wrappers/s2i/python/Makefile b/wrappers/s2i/python/Makefile index 2219dee281..0584cae7f5 100644 --- a/wrappers/s2i/python/Makefile +++ b/wrappers/s2i/python/Makefile @@ -18,6 +18,7 @@ get_wrappers_and_protos: cp $(SELDON_CORE_DIR)/wrappers/python/__init__.py _wrappers/python cp $(SELDON_CORE_DIR)/proto/prediction.proto _wrappers/python/proto cp $(SELDON_CORE_DIR)/wrappers/python/seldon_flatbuffers.py _wrappers/python + cp $(SELDON_CORE_DIR)/wrappers/python/metrics.py _wrappers/python flatc --python -o _wrappers/python/fbs ../../../fbs/prediction.fbs touch _wrappers/python/proto/__init__.py touch _wrappers/python/fbs/__init__.py