updated doc with more notebooks

.gitignore

@@ -161,3 +161,4 @@ examples/wrappers/go/.idea
 wrappers/s2i/python_openvino/s2i
 
 doc/_build
+doc/source/_static/cluster-manager

doc/Makefile

@@ -13,6 +13,28 @@ help:
 
 .PHONY: help Makefile
 
+source/_static/cluster-manager:
+	mkdir -p source/_static/cluster-manager
+
+source/_static/cluster-manager/dependencies: source/_static/cluster-manager
+	cd ../cluster-manager && mvn clean project-info-reports:dependencies
+	mkdir -p source/_static/cluster-manager/dependencies
+	cp -R ../cluster-manager/target/site/* source/_static/cluster-manager/dependencies
+
+source/_static/cluster-manager/jacoco: source/_static/cluster-manager
+	cd ../cluster-manager && mvn clean verify
+	cp -R ../cluster-manager/target/site/jacoco source/_static/cluster-manager
+
+#html:: source/_static/cluster-manager/dependencies source/_static/cluster-manager/jacoco
+
+.PHONY: clean
+clean:
+	rm -r source/_static/cluster-manager/dependencies
+	rm -r source/_static/cluster-manager/jacoco
+
+livehtml:
+	sphinx-autobuild -b html --ignore "*~" $(ALLSPHINXOPTS) $(SOURCEDIR) $(BUILDDIR)/html
+
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile

doc/source/R/README.md

@@ -0,0 +1,157 @@
+# Packaging an R model for Seldon Core using s2i
+
+In this guide, we illustrate the steps needed to wrap your own R model in a docker image ready for deployment with Seldon Core using [source-to-image app s2i](https://github.com/openshift/source-to-image).
+
+If you are not familiar with s2i you can read [general instructions on using s2i](./s2i.md) and then follow the steps below.
+
+## Step 1 - Install s2i
+
+[Download and install s2i](https://github.com/openshift/source-to-image#installation)
+
+- Prerequisites for using s2i are:
+  - Docker
+  - Git (if building from a remote git repo)
+
+To check everything is working you can run
+
+```bash
+s2i usage seldonio/seldon-core-s2i-r:0.1
+```
+
+## Step 2 - Create your source code
+
+To use our s2i builder image to package your R model you will need:
+
+- An R file which provides an S3 class for your model via an `initialise_seldon` function and that has appropriate generics for your component, e.g. predict for a model.
+- An optional install.R to be run to install any libraries needed
+- .s2i/environment - model definitions used by the s2i builder to correctly wrap your model
+
+We will go into detail for each of these steps:
+
+### R Runtime Model file
+
+Your source code should contain an R file which defines an S3 class for your model. For example, looking at our skeleton R model file at `wrappers/s2i/R/test/model-template-app/MyModel.R`:
+
+```R
+library(methods)
+
+predict.mymodel <- function(mymodel,newdata=list()) {
+  write("MyModel predict called", stdout())
+  newdata
+}
+
+
+new_mymodel <- function() {
+  structure(list(), class = "mymodel")
+}
+
+
+initialise_seldon <- function(params) {
+  new_mymodel()
+}
+```
+
+- A `seldon_initialise` function creates an S3 class for my model via a constructor `new_mymodel`. This will be called on startup and you can use this to load any parameters your model needs.
+- A generic `predict` function is created for my model class. This will be called with a `newdata` field with the `data.frame` to be predicted.
+
+There are similar templates for ROUTERS and TRANSFORMERS.
+
+### install.R
+
+Populate an `install.R` with any software dependencies your code requires. For example:
+
+```R
+install.packages('rpart')
+```
+
+### .s2i/environment
+
+Define the core parameters needed by our R builder image to wrap your model. An example is:
+
+```bash
+MODEL_NAME=MyModel.R
+API_TYPE=REST
+SERVICE_TYPE=MODEL
+PERSISTENCE=0
+```
+
+These values can also be provided or overridden on the command line when building the image.
+
+## Step 3 - Build your image
+
+Use `s2i build` to create your Docker image from source code. You will need Docker installed on the machine and optionally git if your source code is in a public git repo.
+
+Using s2i you can build directly from a git repo or from a local source folder. See the [s2i docs](https://github.com/openshift/source-to-image/blob/master/docs/cli.md#s2i-build) for further details. The general format is:
+
+```bash
+s2i build <git-repo> seldonio/seldon-core-s2i-r:0.1 <my-image-name>
+s2i build <src-folder> seldonio/seldon-core-s2i-r:0.1 <my-image-name>
+```
+
+An example invocation using the test template model inside seldon-core:
+
+```bash
+s2i build https://github.com/seldonio/seldon-core.git --context-dir=wrappers/s2i/R/test/model-template-app seldonio/seldon-core-s2i-r:0.1 seldon-core-template-model
+```
+
+The above s2i build invocation:
+
+- uses the GitHub repo: https://github.com/seldonio/seldon-core.git and the directory `wrappers/s2i/R/test/model-template-app` inside that repo.
+- uses the builder image `seldonio/seldon-core-s2i-r`
+- creates a docker image `seldon-core-template-model`
+
+For building from a local source folder, an example where we clone the seldon-core repo:
+
+```bash
+git clone https://github.com/seldonio/seldon-core.git
+cd seldon-core
+s2i build wrappers/s2i/R/test/model-template-app seldonio/seldon-core-s2i-r:0.1 seldon-core-template-model
+```
+
+For more help see:
+
+```
+s2i usage seldonio/seldon-core-s2i-r:0.1
+s2i build --help
+```
+
+## Reference
+
+### Environment Variables
+
+The required environment variables understood by the builder image are explained below. You can provide them in the `.s2i/environment` file or on the `s2i build` command line.
+
+#### MODEL_NAME
+
+The name of the R file containing the model.
+
+#### API_TYPE
+
+API type to create. Can be REST only at present.
+
+#### SERVICE_TYPE
+
+The service type being created. Available options are:
+
+- MODEL
+- ROUTER
+- TRANSFORMER
+
+#### PERSISTENCE
+
+Can only by 0 at present. In future, will allow the state of the component to be saved periodically.
+
+### Creating different service types
+
+#### MODEL
+
+- [A minimal skeleton for model source code](https://github.com/cliveseldon/seldon-core/tree/s2i/wrappers/s2i/R/test/model-template-app)
+- [Example models](https://github.com/SeldonIO/seldon-core/tree/master/examples/models)
+
+#### ROUTER
+- [Description of routers in Seldon Core](../../components/routers/README.md)
+- [A minimal skeleton for router source code](https://github.com/cliveseldon/seldon-core/tree/s2i/wrappers/s2i/R/test/router-template-app)
+
+#### TRANSFORMER
+
+- [A minimal skeleton for transformer source code](https://github.com/cliveseldon/seldon-core/tree/s2i/wrappers/s2i/R/test/transformer-template-app)

doc/source/analytics/analytics.md

@@ -0,0 +1,54 @@
+# Seldon Core Analytics
+
+Seldon Core exposes metrics that can be scraped by Prometheus. The core metrics are exposed by the service orchestrator (```engine```) and API gateway (```server_ingress```).
+
+The metrics are:
+
+Prediction Requests
+
+ * ```seldon_api_engine_server_requests_duration_seconds_(bucket,count,sum) ``` : Requests to the service orchestrator from an ingress, e.g. API gateway or Ambassador
+ * ```seldon_api_engine_client_requests_duration_seconds_(bucket,count,sum) ``` : Requests from the service orchestrator to a component, e.g., a model
+ * ```seldon_api_server_ingress_requests_duration_seconds_(bucket,count,sum) ``` : Requests to the API Gateway from an external client
+
+Feedback Requests
+
+ * ```seldon_api_model_feedback_reward_total``` : Reward sent via Feedback API
+ * ```seldon_api_model_feedback_total``` : Total feedback requests
+
+Each metric has the following key value pairs for further filtering which will be taken from the SeldonDeployment custom resource that is running:
+
+  * deployment_name
+  * predictor_name
+  * predictor_version
+    * This will be derived from the predictor metadata labels
+  * model_name
+  * model_image
+  * model_image
+
+
+## Helm Analytics Chart
+
+Seldon Core provides an example Helm analytics chart that displays the above Prometheus metrics in Grafana. You can install it with:
+
+```
+helm install seldon-core-analytics --name seldon-core-analytics \
+     --repo https://storage.googleapis.com/seldon-charts \
+     --set grafana_prom_admin_password=password \
+     --set persistence.enabled=false \
+```
+
+The available parameters are:
+
+ * ```grafana_prom_admin_password``` : The admin user Grafana password to use.
+ * ```persistence.enabled``` : Whether Prometheus persistence is enabled.
+
+Once running you can expose the Grafana dashboard with:
+
+```
+kubectl port-forward $(kubectl get pods -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') 3000:3000
+```
+
+You can then view the dashboard at http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1
+
+![dashboard](./dashboard.png)
+

doc/source/analytics/custom_metrics.md

@@ -0,0 +1,70 @@
+# Custom Metrics
+
+Seldon Core exposes basic metrics via Prometheus endpoints on its service orchestrator that include request count, request time percentiles and rolling accuracy for each running model. However, you may wish to expose custom metrics from your components which are automaticlaly added to Prometheus. For this purpose you can supply extra fields in the returned meta data of the response object in the API calls to your components as illustrated below:
+
+```
+{
+	"meta": {
+		"metrics": [
+			{
+				"type": "COUNTER",
+				"key": "mycounter",
+				"value": 1.0
+				"tags": {"mytag":"mytagvalue"}
+			},
+			{
+				"type": "GAUGE",
+				"key": "mygauge",
+				"value": 22.0
+			},
+			{
+				"type": "TIMER",
+				"key": "mytimer",
+				"value": 1.0
+			}
+		]
+	},
+	"data": {
+		"ndarray": [
+			[
+				1,
+				2
+			]
+		]
+	}
+}
+```
+
+We provide three types of metric that can be returned in the meta.metrics list:
+
+ * COUNTER : a monotonically increasing value. It will be added to any existing value from the metric key.
+ * GAUGE : an absolute value showing a level, it will overwrite any existing value.
+ * TIMER : a time value (in msecs)
+
+Each metric apart from the type takes a key and a value. The proto buffer definition is shown below:
+
+```
+message Metric {
+ enum MetricType {
+     COUNTER = 0;
+     GAUGE = 1;
+     TIMER = 2;
+ }
+ string key = 1;
+ MetricType type = 2;
+ float value = 3;
+ map<string,string> tags = 4; 
+}
+```
+
+
+As we expose the metrics via Prometheus, if ```tags``` are added they must appear in every metric response and always have the same set of keys as Prometheus does not allow metrics to have varying numbers of tags. This condition is enforced by the [micrometer](https://micrometer.io/) library we use to expose the metrics and exceptions will happen if violated.
+
+At present the following Seldon Core wrappers provide integrations with custom metrics:
+
+ * [Python Wrapper](./wrappers/python.md#custom-metrics)
+
+
+## Example
+
+There is an [example notebook illustrating a model with custom metrics in python](../examples/models/template_model_with_metrics/modelWithMetrics.ipynb).

doc/source/analytics/outlier_detection.rst

@@ -0,0 +1,3 @@
+
+.. mdinclude:: ../../../components/outlier-detection/README.md
+

doc/source/conf.py

@@ -118,7 +118,9 @@
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+   'sticky_navigation': False
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

doc/source/developer/build-using-private-repo.md

@@ -0,0 +1,61 @@
+# Build using private local repository
+
+## Prerequisites
+
+* Local Docker
+* Kubernetes cluster access
+* Private local repository setup in the cluster with local access
+    * use the project [k8s-local-docker-registry](https://github.com/SeldonIO/k8s-local-docker-registry)
+    * "127.0.0.1:5000" will be used as the repo host url
+
+## Prerequisite check
+
+Ensure the prerequisites are in place and the correct ports available.
+
+```
+# Check that the private local registry works
+(set -x && curl -X GET http://127.0.0.1:5000/v2/_catalog && \
+        docker pull busybox && docker tag busybox 127.0.0.1:5000/busybox && \
+        docker push 127.0.0.1:5000/busybox)
+```
+
+## Updating components and redeploying into cluster
+
+Basic process of how to test code changes in cluster.
+
+1. Stop seldon core if its running.
+1. Build and push the component that was updated or all components if necessary.
+1. Start seldon core.
+1. Deploy models.
+
+Below are details to achieve this.
+
+### Building all components
+
+Build all images and push to private local repository.
+
+```
+./build-all-private-repo
+./push-all-private-repo
+```
+
+### start/stop Seldon Core
+
+```
+./start-seldon-core-private-repo
+./stop-seldon-core-private-repo
+```
+
+### Building individual components
+
+```
+./cluster-manager/build-private-repo
+./cluster-manager/push-private-repo
+
+./api-frontend/build-private-repo
+./api-frontend/push-private-repo
+
+./engine/build-private-repo
+./engine/push-private-repo
+```
+