Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update v0.2.0 branch #365

Merged
merged 21 commits into from
Oct 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e03034f
Add dev doc links to CONTRIBUTING.md (#312)
rafvasq Mar 11, 2022
a5d6177
Fixes the references to latest kfctl release (#311)
rafvasq Mar 16, 2022
07377ff
Do not verify links in project dependencies (#320)
ckadner Mar 18, 2022
e26ce79
Add navigation and description table of docs (#314)
rafvasq Mar 18, 2022
e3bfcc7
Add script to update the docs table (#317)
rafvasq Mar 30, 2022
3671efa
Modified the way sed version is set (#315)
krishnakumar27 Mar 30, 2022
8b8e639
Bump waitress from 2.0.0 to 2.1.1 in /api/server (#321)
dependabot[bot] Mar 30, 2022
262f26b
Bump jupyter-server from 1.13.4 to 1.15.4 in /api/server (#324)
dependabot[bot] Mar 30, 2022
e566b20
Bump minimist from 1.2.5 to 1.2.6 in /dashboard/origin-mlx (#326)
dependabot[bot] Mar 30, 2022
ef85c69
Bump notebook from 6.4.8 to 6.4.10 in /api/server (#327)
dependabot[bot] Apr 8, 2022
fe965be
Update API developer docs (#325)
shanmukha1996 Mar 30, 2022
605fe33
Update UI developer docs (#323)
RRM123 Apr 22, 2022
3fc0338
Correct description for make update_doc_table (#329)
ezinneanne May 5, 2022
7fee873
Update Kubernetes high version in deployment docs (#318) (#332)
kiranp2396 May 25, 2022
5f41510
fix errors in mlx-ui startup (#338)
jbusche Jun 13, 2022
d78055e
Run mlx-ui as non-root user (#339)
ckadner Jun 15, 2022
6680f55
Update MLX setup instructions for KF 1.5 (#346)
ckadner Jun 24, 2022
5baeef8
fix selected assets show up in different menus (#342)
JiaxuanYang Jun 24, 2022
5cdaf07
Consolidate readme (#356)
rafvasq Oct 3, 2022
5e9958f
Document MLX Models Workshop (#352)
ckadner Oct 3, 2022
dc48b0f
Add GitHub action to verify doc links (#357)
rafvasq Oct 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/verify-links.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Copyright 2021 The MLX Contributors
#
# SPDX-License-Identifier: Apache-2.0

name: Verify Links

on:
push:
branches:
- main

tags:
- v*

pull_request:

jobs:
check-links:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]
steps:
- name: Checkout
uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: pip install requests

- name: Link check
run: |
./tools/python/verify_doc_links.py

16 changes: 8 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,14 @@ maintainers of each component affected.

For a list of the maintainers, see the [MAINTAINERS.md](MAINTAINERS.md) page.

## Getting Started

**MLX Dashboard Web UI**
- Information regarding configuration, deployment, and development setup can be found [here](https://github.com/machine-learning-exchange/mlx/blob/main/dashboard/origin-mlx/README.md).

**MLX API**
- Quickstart and development setup documentation can be found [here](https://github.com/machine-learning-exchange/mlx/tree/main/api).

## Legal

Each source file must include a license header for the Apache
Expand Down Expand Up @@ -72,14 +80,6 @@ git commit -s
## Communication
Please feel free to connect with us on our [Slack channel](https://lfaifoundation.slack.com/archives/C0264LKNH63).

## Setup
**FIXME** Please add any special setup instructions for your project to help the developer
become productive quickly.

## Testing
**FIXME** Please provide information that helps the developer test any changes they make
before submitting.

## Coding style guidelines
**FIXME** Optional, but recommended: please share any specific style guidelines you might
have for your project.
Expand Down
5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ check_doc_links: ## Check markdown files for invalid links
@python3 tools/python/verify_doc_links.py
@echo "$@: OK"

.PHONY: update_doc_table
update_doc_table: ## Regenerate the /docs/README.md file
@python3 tools/python/update_doc_table.py
@echo "$@: OK"

.PHONY: check_license
check_license: ## Make sure source files have license header
@git grep -L "SPDX-License-Identifier: Apache-2.0" -- *.py *.yml *.yaml *.sh *.html *.js *.css *.ts *.tsx ':!*.bundle.js' | \
Expand Down
31 changes: 15 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,6 @@ Allows upload, registration, execution, and deployment of:
- Datasets
- Notebooks

For more details about the project please follow this [announcement blog post](https://lfaidata.foundation/blog/2021/09/28/machine-learning-exchange-mlx/).


<img src="docs/images/mlx.png" height="90%" width="90%">

Additionally it provides:

- Automated sample pipeline code generation to execute registered models, datasets and notebooks
Expand All @@ -28,6 +23,12 @@ Additionally it provides:
- Serving engine by [KFServing](https://github.com/kubeflow/kfserving)
- Model Metadata schemas

For more details about the project check out this [announcement blog post](https://lfaidata.foundation/blog/2021/09/28/machine-learning-exchange-mlx/).


<img src="docs/images/mlx.png" height="90%" width="90%">


## 1. Deployment
<img src="docs/images/mlx-architecture-4.png" height="40%" width="40%">

Expand All @@ -48,30 +49,28 @@ For a full deployment, we use [Kubeflow Kfctl](https://github.com/kubeflow/kfctl
* #### [MLX on an existing Kubeflow Cluster](./docs/install-mlx-on-kubeflow.md)


## 2. Access the MLX UI
## 2. Access the MLX UI and import Assets to the Catalog

By default, the MLX UI is available at http://<cluster_node_ip>:30380/mlx/

If you deployed on a **Kubernetes** cluster, run the following and look for the External-IP column to find the public IP of a node.
If you deployed on a **Kubernetes** cluster or using **OpenShift**, run the following and look for the External-IP column to find the public IP of a node.

```bash
kubectl get node -o wide
```

If you deployed using **OpenShift**, you can use IstioIngresGateway Route. You can find it in the OpenShift Console or using the CLI.
For information on how to import data and AI assets using MLX's catalog importer, use this [guide](/docs/import-assets.md).

```bash
oc get route -n istio-system
```
## 3. Use MLX

## 3. Import Data and AI Assets in MLX Catalog
For information on how to use MLX and create assets check out this [guide](/docs/usage-steps.md).

[Import data and AI assets using MLX's catalog importer](/docs/import-assets.md)
## 4. How to Contribute

For information about adding new features, bug fixing, communication
or UI and API setup, refer to this [document](CONTRIBUTING.md).

## 4. Use MLX

[MLX Usage Instructions](/docs/usage-steps.md)

## 5. Troubleshooting

[MLX Troubleshooting Instructions](/docs/troubleshooting.md)
Expand Down
76 changes: 60 additions & 16 deletions api/README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,59 @@
# Machine Learning Exchange API - Python Client and Python-Flask Server
# MLX API - Python Client and Python-Flask Server

An extension to the Kubeflow Pipeline API for Components and Models
The MLX API is an extension to the Kubeflow Pipeline API with additional API
endpoints for Dataset, Models, Notebooks and Pipeline Components.
We use [OpenAPI v2 (fka Swagger)](https://swagger.io/specification/v2/) for the
[API specification](swagger/swagger.yaml).

---

# Quickstart
# Deploy to Kubernetes

## Deploy to Kubernetes
If you already have a Kubeflow Pipelines deployment on a Kubernetes cluster, you
can use the following steps to deploy the MLX API on top of it. However, for a full
deployment of MLX we recommend following on of these [guides](../README.md#1-deployment)

kubectl apply -f ./server/mlx-api.yml
1) Run kubectl command to apply the manifest

## Find API Server Host and Port
`kubectl apply -f ./server/mlx-api.yml`

export API_HOST=$(kubectl get nodes -o jsonpath='{.items[].status.addresses[?(@.type=="ExternalIP")].address}')
export API_PORT=$(kubectl get service mlx-api -n kubeflow -o jsonpath='{.spec.ports[0].nodePort}')
2) Find API Server Host and Port

## Open the Swagger UI in a Web Browser
`export API_HOST=$(kubectl get nodes -o jsonpath='{.items[].status.addresses[?(@.type=="ExternalIP")].address}')`
`export API_PORT=$(kubectl get service mlx-api -n kubeflow -o jsonpath='{.spec.ports[0].nodePort}')`

open "http://${API_HOST}:${API_PORT}/apis/v1alpha1/ui/"
3) Open the Swagger UI in a Web Browser

`open "http://${API_HOST}:${API_PORT}/apis/v1alpha1/ui/" `


---

# Development Setup
# API Development

## Code Generation Overview

![API Codegeneration Workflow](codegen_workflow.png)

- Changes/additions to the API are done in the API [spec](swagger/swagger.yaml)
- The [`generate_code.sh`](generate_code.sh) script validates the API [spec](swagger/swagger.yaml)
and generates [`client`](client) and [`server`](server) Python packages.
The [examples](examples) package is hand-written based on the usage examples
inside the `client` package
- Instead of a static HTML documentation for the API endpoints, a Swagger UI web
interface will be generated on the fly when the Python server is started
- The API endpoints and the JSON data model is documented under [`client/docs`](client/docs)
in Markdown format which works well when browsing through the GitHub repo
- After the code generation, we need to copy any new API method stubs (if any new)
from the [`server/swagger_server/controllers`](server/swagger_server/controllers) folder
to the [`server/swagger_server/controllers_impl`](server/swagger_server/controllers_impl)
folder and implement the actual business logic
- If existing API method signatures got updated, we need to update the existing
`controller_impl` methods respectively


## Swagger Codegen 2.4
## Development Setup

### Swagger Codegen 2.4

To generate our API we are using [`swagger-codegen`](https://github.com/swagger-api/swagger-codegen/tree/v2.4.8#prerequisites)
version [`2.4`](https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.4.8/swagger-codegen-cli-2.4.8.jar)
Expand All @@ -48,20 +78,29 @@ will automatically download the _"correct"_ version of the `swagger-codegen-cli.
# brew install swagger-codegen@2
# brew link --force swagger-codegen@2

## Create a Python Virtual Environment for Development
### Create a Python Virtual Environment for Development

python3 -m venv .venv
source .venv/bin/activate

## Install the Python Package Dependencies
### Install the Python Package Dependencies

# cd <mlx_root_dir>
# cd api

pip install -r ./requirements.txt



## (Re-)Generate Swagger Client and Server Code

If there are changes to the [API spec](swagger/swagger.yaml) then we need to re-
generate the API client and server code. Do not run the script `codegen.sh`
on its own, since Swagger generates a lot of unwanted and some breaking changes.
Instead use the [generate_code.sh](generate_code.sh) script which runs the Swagger
codegen and tries to undo some of the code and documentation changes that are not
desired.

./generate_code.sh

## Build the Docker Image
Expand All @@ -86,7 +125,8 @@ or:
kubectl delete -f ./server/mlx-api.yml
kubectl apply -f ./server/mlx-api.yml

## Testing API Code Changes with Docker Compose

## Testing API Code Changes Locally with Docker Compose

You can test most code changes without a Kubernetes cluster. A K8s cluster is only
required to `run` the generated sample pipeline code. Running the Quickstart with
Expand Down Expand Up @@ -189,3 +229,7 @@ Docker Compose stack with the `-v` option (`docker compose --project-name no_api
-d @catalog_upload.json \
http://localhost:8080/apis/v1alpha1/catalog

# Troubleshooting

Report any issues with at https://github.com/machine-learning-exchange/mlx/issues

17 changes: 6 additions & 11 deletions api/codegen.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,13 @@ swagger-codegen generate -i swagger/swagger.yaml -l python-flask -o server 2>&1
# set interactive mode to enable defining a gsed alias
shopt -s expand_aliases

# we use sed to make in-file text replacements, but sed works differently on macOS and Linux
if [[ "$OSTYPE" == "linux-gnu"* ]]; then # Linux
alias gsed="sed -i"
elif [[ "$OSTYPE" == "darwin"* ]]; then # macOS
alias gsed="sed -i ''"
elif [[ "$OSTYPE" == "cygwin" ]]; then # POSIX compatible emulation for Windows
alias gsed="sed -i"
elif [[ "$OSTYPE" == "msys"* ]]; then # Git Bash (Windows)
alias gsed="sed -i"
# we use sed to make in-file text replacements, but sed works differently depending on the version
if ! sed -i '1s/^/test/' $(mktemp) 2> /dev/null; then
# macOS (BSD) version of sed
alias gsed="sed -i ''"
else
echo "FAILED. OS not compatible with script '${BASH_SOURCE[0]}'"
exit 1
# POSIX compliant version of sed
alias gsed="sed -i"
fi
export gsed

Expand Down
Binary file added api/codegen_workflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 18 additions & 2 deletions api/server/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,29 @@
# Swagger generated server
# MLX API Server

## Overview
This server was generated by the [swagger-codegen](https://github.com/swagger-api/swagger-codegen)

The MLX API server was generated by the [swagger-codegen](https://github.com/swagger-api/swagger-codegen)
project. By using the [OpenAPI-Spec](https://github.com/swagger-api/swagger-core/wiki) from a remote
server, you can easily generate a server stub. This is an example of building a Swagger-enabled
Flask server.

This example uses the [Connexion](https://github.com/zalando/connexion) library on top of Flask.

## MLX API Server Modules

| Python package/script | Purpose |
| ------------- | ------------- |
| `code_templates` | Code templates are used to generate sample Kubeflow pipeline DSL scripts for each asset type |
| `controllers` | Controllers are the code stubs (“interfaces”) for the API endpoints generated by Swagger |
| `controllers_impl` | Controller implementation of the generated API code stubs (“interfaces”), originally copied from respective controller and then filled in with the actual business logic. API calls get forwarded from the `controllers` to the `controllers_impl` by injection of some code magic in `util.py` (see `codegen.sh`) |
| `data_access` | Contains 2 clients - MySQL (relational database) and Minio (object storage) for data access |
| `gateways` | Connects the API server to various services on the Kubernetes cluster like KFServing, Kubeflow Pipelines, and the Kubernetes API |
| `models` | Swagger generated API classes used to transfer data between API client and API server |
| `encoder.py` | Implementation of JSON encoder (generated by Swagger) |
| `util.py` | Collection of helper functions, most notably to “glue” the API `controllers` code stubs to the `controllers_impl` API implementations |

## Requirements

Python 3.6.1+

## Usage
Expand All @@ -33,6 +48,7 @@ http://localhost:8080/apis/v1alpha1/swagger.json
```

To launch the integration tests, use tox:

```
sudo pip install tox
tox
Expand Down
17 changes: 5 additions & 12 deletions api/server/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,6 @@ ansiwrap==0.8.4
# via papermill
anyio==3.5.0
# via jupyter-server
appnope==0.1.2
# via
# ipykernel
# ipython
argon2-cffi==21.3.0
# via
# jupyter-server
Expand Down Expand Up @@ -188,10 +184,7 @@ ipython==8.0.1
# ipykernel
# jupyterlab
ipython-genutils==0.2.0
# via
# jupyter-server
# nbformat
# notebook
# via notebook
isodate==0.6.1
# via openapi-schema-validator
isort==5.10.1
Expand Down Expand Up @@ -244,7 +237,7 @@ jupyter-lsp==1.5.1
# via jupyterlab-lsp
jupyter-resource-usage==0.6.1
# via elyra-server
jupyter-server==1.13.4
jupyter-server==1.15.4
# via
# elyra-server
# jupyter-lsp
Expand Down Expand Up @@ -327,7 +320,7 @@ nbdime==3.1.1
# via
# elyra-server
# jupyterlab-git
nbformat==5.1.3
nbformat==5.2.0
# via
# elyra-server
# jupyter-server
Expand All @@ -345,7 +338,7 @@ nest-asyncio==1.5.4
# notebook
networkx==2.6.3
# via elyra-server
notebook==6.4.8
notebook==6.4.10
# via nbclassic
numpy==1.22.1
# via
Expand Down Expand Up @@ -612,7 +605,7 @@ urllib3==1.26.8
# kubernetes
# minio
# requests
waitress==2.0.0
waitress==2.1.1
# via -r requirements.in
watchdog==2.1.6
# via elyra-server
Expand Down
Loading
Loading