Merge branch 'master' of https://github.com/kubeflow/website into docsy

OWNERS

@@ -11,6 +11,7 @@ approvers:
   - sarahmaddox
   - texasmichelle
   - willingc
+  - dsdinter
 reviewers:
   - abhi-g
   - aronchick
@@ -25,3 +26,4 @@ reviewers:
   - sarahmaddox
   - texasmichelle
   - willingc
+  - dsdinter

config.toml

@@ -186,8 +186,14 @@ breadcrumb_disable = false
   identifier = "components"
   url = "/docs/guides/components/"
 [[menu.docs]]
-  name = "GKE"
+  name = "Pipelines"
   weight = 5
   parent = "guides"
+  identifier = "pipelines"
+  url = "/docs/guides/pipelines/"
+[[menu.docs]]
+  name = "GKE"
+  weight = 6
+  parent = "guides"
   identifier = "gke"
   url = "/docs/guides/gke/"

content/docs/about/community.md

@@ -52,7 +52,7 @@ mailing list.
 
 Summary:
 
-* [Invitation to our Slack channel](https://join.slack.com/t/kubeflow/shared_invite/enQtMjgyMzMxNDgyMTQ5LWUwMTIxNmZlZTk2NGU0MmFiNDE4YWJiMzFiOGNkZGZjZmRlNTExNmUwMmQ2NzMwYzk5YzQxOWQyODBlZGY2OTg)
+* [Invitation to our Slack channel](https://join.slack.com/t/kubeflow/shared_invite/enQtNDg5MTM4NTQyNjczLWUyZGI1ZmExZWExYWY4YzlkOWI4NjljNjJhZjhjMjEwNGFjNmVkNjg2NTg4M2I0ZTM5NDExZWI5YTIyMzVmNzM)
 * [Twitter](http://twitter.com/kubeflow)
 * [Mailing list: kubeflow-discuss](https://groups.google.com/forum/#!forum/kubeflow-discuss)
 

content/docs/guides/components/mpi.md

@@ -7,7 +7,7 @@ bref= "This guide will walk you through using MPI for training"
 [menu]
 [menu.docs]
   parent = "components"
-  weight = 30
+  weight = 25
 +++
 
 ## Installing MPI Operator

content/docs/guides/components/mxnet.md

@@ -7,7 +7,7 @@ bref= "This guide will walk you through using MXNet with Kubeflow"
 [menu]
 [menu.docs]
   parent = "components"
-  weight = 30
+  weight = 25
 +++
 
 ## Installing MXNet Operator

content/docs/guides/components/pipelines.md

@@ -0,0 +1,26 @@
++++
+title = "Pipelines"
+description = "Introduction to Kubeflow Pipelines"
+weight = 10
+toc = true
+[menu]
+[menu.docs]
+  parent = "components"
+  weight = 35
++++
+
+Kubeflow Pipelines is a platform for building and deploying portable and
+scalable end-to-end ML workflows, based on containers.
+
+The Kubeflow Pipelines platform has the following goals:
+
+* End-to-end orchestration: enabling and simplifying the orchestration of
+  machine learning pipelines.
+* Easy experimentation: making it easy for you to try numerous ideas and 
+  techniques and manage your various trials/experiments.
+* Easy re-use: enabling you to re-use components and pipelines to quickly 
+  cobble together end-to-end solutions, without having to rebuild each time.
+
+  Read more in the 
+  [pipelines section](/docs/guides/pipelines/pipelines-overview)
+  of the documentation.

content/docs/guides/components/pytorch.md

@@ -7,7 +7,7 @@ bref= "This guide will walk you through using PyTorch with Kubeflow"
 [menu]
 [menu.docs]
   parent = "components"
-  weight = 30
+  weight = 35
 +++
 
 ## Installing PyTorch Operator

content/docs/guides/components/pytorchserving.md

@@ -0,0 +1,79 @@
++++
+title = "PyTorch Serving"
+description = "Instructions for serving a PyTorch model with Seldon"
+weight = 10
+toc = true
+bref= "This guide will walk you through serving a PyTorch trained model in Kubeflow"
+[menu]
+[menu.docs]
+  parent = "components"
+  weight = 35
++++
+
+## Serving a model
+
+We use [seldon-core](https://github.com/SeldonIO/seldon-core) component deployed following [these](/docs/guides/components/seldon/) instructions to serve the model.
+
+See also this [Example module](https://github.com/kubeflow/examples/blob/master/pytorch_mnist/serving/seldon-wrapper/mnistddpserving.py) which contains the code to wrap the model with Seldon. 
+
+We will wrap this class into a seldon-core microservice which we can then deploy as a REST or GRPC API server.
+
+##  Building a model server
+
+We use the public model server image `gcr.io/kubeflow-examples/mnistddpserving` as an example
+
+  * This server loads the model from the mount point /mnt/kubeflow-gcfs and includes the supporting assets baked into the container image
+  * So you can just run this image to get a pre-trained model from the shared persistent disk
+  * Serving your own model using this server, exposing predict service as GRPC API
+
+## Building your own model server
+
+You can use the below command to build your own image to wrap your model, also check [this](https://github.com/kubeflow/examples/blob/master/pytorch_mnist/serving/seldon-wrapper/build_image.sh) 
+script example that calls the docker Seldon wrapper to build our server image, exposing the predict service as GRPC API.
+```
+docker run -v $(pwd):/my_model seldonio/core-python-wrapper:0.7 /my_model mnistddpserving 0.1 gcr.io --image-name=kubeflow-examples/mnistddpserving --grpc
+```
+
+You can then push the image by running `gcloud docker -- push gcr.io/kubeflow-examples/mnistddpserving:0.1`.
+
+You can find more details about wrapping a model with seldon-core [here](https://github.com/SeldonIO/seldon-core/blob/master/docs/wrappers/python.md)
+
+## Deploying the model to your Kubeflow cluster
+
+We need to have seldon component deployed, you can deploy the model once trained using a pre-defined ksonnet component, similar to [this](https://github.com/kubeflow/examples/blob/master/pytorch_mnist/ks_app/components/serving_model.jsonnet) example.
+We need to setup our own environment `${KF_ENV}` (e.g., 'default') and modify the Ksonnet component 
+[parameters](https://github.com/kubeflow/examples/blob/master/pytorch_mnist/ks_app/components/params.libsonnet) to use your specific image.
+
+```bash
+cd ks_app
+ks env add ${KF_ENV}
+ks apply ${KF_ENV} -c serving_model
+```
+
+## Testing model server
+
+Seldon Core component uses ambassador to route it's requests to our model server. To send requests to the model, you can port-forward the ambassador container locally:
+
+```
+kubectl port-forward $(kubectl get pods -n ${NAMESPACE} -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n ${NAMESPACE} 8080:80
+```
+
+And send a request, for our example we know is not a torch MNIST image, so it will return an error 500
+
+```
+curl -X POST -H 'Content-Type: application/json' -d '{"data":{"int":"8"}}' http://localhost:8080/seldon/mnist-classifier/api/v0.1/predictions
+```
+
+We should receive an error response as the model server is expecting a 1x786 vector representing a torch image, this will be sufficient to confirm the server model is up and running
+(This is to avoid having to send manually a vector of 786 pixels, you can interact properly with the model using a web interface if you follow all the 
+[instructions](https://github.com/kubeflow/examples/tree/master/pytorch_mnist) in the example)
+
+```
+{
+"timestamp":1540899355053,
+"status":500,"error":"Internal Server Error",
+"exception":"io.grpc.StatusRuntimeException",
+"message":"UNKNOWN: Exception calling application: tensor is not a torch image.",
+"path":"/api/v0.1/predictions"
+}
+```

content/docs/guides/components/tfserving_new.md

@@ -61,8 +61,6 @@ See [doc](https://cloud.google.com/docs/authentication/) for more detail.
 
 *S3*
 
-TODO(https://github.com/kubeflow/kubeflow/issues/1309): fix aws part
-
 To use S3, generate a different prototype
 ```
 ks generate tf-serving-aws ${MODEL_COMPONENT} --name=${MODEL_NAME}

content/docs/guides/components/tftraining.md

@@ -1,10 +1,10 @@
 +++
 title = "TensorFlow Training (TFJob)"
-linkTitle = "TensorFlow Training"
+linkTitle = "TensorFlow Training (TFJob)"
 description = ""
 weight = 10
 toc = true
-bref= "Training using TensorFlow"
+bref= "This page describes TFJob for training a machine learning model with TensorFlow."
 [menu]
 [menu.docs]
   parent = "components"
@@ -13,8 +13,10 @@ bref= "Training using TensorFlow"
 
 ## What is TFJob?
 
-TFJob is a Kubernetes [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) that makes it easy to
-run TensorFlow training jobs on Kubernetes.
+TFJob is a Kubernetes 
+[custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) that you can use to run TensorFlow training jobs on Kubernetes. The Kubeflow
+implementation of TFJob is in
+[`tf-operator`](https://github.com/kubeflow/tf-operator).
 
 A TFJob is a resource with a simple YAML representation illustrated below.
 

content/docs/guides/pipelines/build-component.md

@@ -0,0 +1,174 @@
++++
+title = "Build Your Own Pipeline Components"
+description = "Building your own components for Kubeflow Pipelines."
+weight = 3
+toc = true
+
+[menu.docs]
+  parent = "pipelines"
+  weight = 6
++++
+This page is for advanced users. It describes how to build your own pipeline 
+components. For an easier start, try 
+[building a pipeline with the provided samples](/docs/guides/pipelines/build-pipeline).
+
+## Overview of pipeline components
+
+Kubeflow Pipelines components are implementations of pipeline tasks. Each task 
+takes one or more 
+[artifacts](/docs/guides/pipelines/pipelines-concepts#step-output-artifacts) as
+input and may produce one or more
+[artifacts](/docs/guides/pipelines/pipelines-concepts#step-output-artifacts) as 
+output.
+
+Each task usually includes two parts:
+
+``Client code``
+  The code that talks to endpoints to submit jobs. For example, code to talk to 
+  the Google Dataproc API to submit a Spark job.
+
+``Runtime code``
+  The code that does the actual job and usually runs in the cluster. For 
+  example, Spark code that transforms raw data into preprocessed data.
+
+Note the naming convention for client code and runtime code—for a task 
+named "mytask":
+
+* The `mytask.py` program contains the client code.
+* The `mytask` directory contains all the runtime code.
+
+A component consists of an interface (inputs/outputs), the implementation 
+(a Docker container image and command-line arguments) and metadata 
+(name, description).
+
+Components can be instantiated inside the `pipeline` function to create tasks.
+
+There are multiple ways to author components:
+
+* Wrap an existing Docker container image using `ContainerOp`, as described 
+  below.
+* Create a 
+  [lightweight python component](/docs/guides/pipelines/lightweight-python-components) 
+  from a Python function
+* Build a new Docker container image from a Python function.
+
+## Example: XGBoost DataProc components
+
+* [Set up cluster](https://github.com/kubeflow/pipelines/blob/master/components/dataproc/xgboost/create_cluster.py)
+* [Analyze](https://github.com/kubeflow/pipelines/blob/master/components/dataproc/xgboost/analyze.py)
+* [Transform](https://github.com/kubeflow/pipelines/blob/master/components/dataproc/xgboost/transform.py)
+* [Train (distributed)](https://github.com/kubeflow/pipelines/blob/master/components/dataproc/xgboost/train.py)
+* [Delete cluster](https://github.com/kubeflow/pipelines/blob/master/components/dataproc/xgboost/delete_cluster.py)
+
+## Requirements for building a component
+
+Install [Docker](https://www.docker.com/get-docker).
+
+## Step One: Create a container for each component
+
+In most cases, you need to create your own container image that includes your 
+program. See the  
+[container building examples](https://github.com/kubeflow/pipelines/blob/master/components). 
+(In the directory, go to any subdirectory and then go to the `containers` directory.)
+
+If your component creates some outputs to be fed as inputs to the downstream 
+components, each output must be a string and must be written to a separate local 
+text file by the container image. For example, if a trainer component needs to 
+output the trained model path, it writes the path into a  local file 
+`/output.txt`. In the Python class (in step three), you have the chance to 
+specify how to map the content  of local files to component outputs.
+
+<!---[TODO]: Add how to produce UI metadata.--->
+
+## Step Two: Create a Python class for your component
+
+The Python classes describe the interactions with the Docker container image 
+created in step one. For example, a component to create confusion matrix data 
+from prediction results looks like this:
+
+```python
+class ConfusionMatrixOp(kfp.dsl.ContainerOp):
+
+  def __init__(self, name, predictions, output_path):
+    super(ConfusionMatrixOp, self).__init__(
+      name=name,
+      image='gcr.io/project-id/ml-pipeline-local-confusion-matrix:v1',
+      command=['python', '/ml/confusion_matrix.py'],
+      arguments=[
+        '--output', '%s/{{workflow.name}}/confusionmatrix' % output_path,
+        '--predictions', predictions
+     ],
+     file_outputs={'label': '/output.txt'})
+
+```
+
+Note:
+
+* Each component needs to inherit from `kfp.dsl.ContainerOp`.
+* If you already defined ENTRYPOINT in the container image, you don’t need to 
+  provide `command` unless you want to override it.
+* In the init arguments, there can be Python native types (such as str, int) and 
+  `kfp.dsl.PipelineParam` types. Each `kfp.dsl.PipelineParam` represents a 
+  parameter whose value is usually only known at run time. It might be a 
+  pipeline  parameter whose value is provided at pipeline run time by user, or 
+  it can be an output from an upstream component. 
+  In the above case, `predictions` and `output_path` are `kfp.dsl.PipelineParam` types.
+* Although the value of each PipelineParam is only available at run time, you 
+  can still use the parameters inline in the  argument (note the “%s”). This 
+  means at run time the argument contains the value of the param inline.
+* `file_outputs` lists a map between labels and local file paths. In the above 
+  case, the content of `/output.txt` is gathered as a string output of the 
+  operator. To reference the output in code:
+
+    ```python
+    op = ConfusionMatrixOp(...)
+    op.outputs['label']
+    ```
+
+If there is only one output then you can also do `op.output`.
+
+## Step Three: Create your workflow as a Python function
+
+Each pipeline is identified as a Python function. For example:
+
+```python
+@kfp.dsl.pipeline(
+  name='TFX Trainer',
+  description='A trainer that does end-to-end training for TFX models.'
+)
+def train(
+    output_path,
+    train_data=kfp.dsl.PipelineParam('train-data',
+        value='gs://ml-pipeline-playground/tfx/taxi-cab-classification/train.csv'),
+    eval_data=kfp.dsl.PipelineParam('eval-data',
+        value='gs://ml-pipeline-playground/tfx/taxi-cab-classification/eval.csv'),
+    schema=kfp.dsl.PipelineParam('schema',
+        value='gs://ml-pipeline-playground/tfx/taxi-cab-classification/schema.json'),
+    target=kfp.dsl.PipelineParam('target', value='tips'),
+    learning_rate=kfp.dsl.PipelineParam('learning-rate', value=0.1),
+    hidden_layer_size=kfp.dsl.PipelineParam('hidden-layer-size', value='100,50'),
+    steps=kfp.dsl.PipelineParam('steps', value=1000),
+    slice_columns=kfp.dsl.PipelineParam('slice-columns', value='trip_start_hour'),
+    true_class=kfp.dsl.PipelineParam('true-class', value='true'),
+    need_analysis=kfp.dsl.PipelineParam('need-analysis', value='true'),
+)
+```
+
+Note:
+
+* **@kfp.dsl.pipeline** is a required decoration including `name` and 
+  `description` properties.
+* Input arguments show up as pipeline parameters in the Kubeflow Pipelines UI. 
+  As a Python rule, positional  args go first and keyword args go next.
+* Each function argument is of type `kfp.dsl.PipelineParam`. The default values 
+  should all be of that type. The default values show up in the Kubeflow 
+  Pipelines UI but can be overwritten.
+
+
+See [an example](https://github.com/kubeflow/pipelines/blob/master/samples/xgboost-spark/xgboost-training-cm.py).
+
+## Lightweight Python components
+
+You can also build lightweight components from Python functions. See the guide 
+to 
+[lightweight python components](/docs/guides/pipelines/lightweight-python-components).