Merge branch 'main' into jorgeorpinel-patch-1

.github/workflows/check-test-release.yml

@@ -88,6 +88,8 @@ jobs:
         HEROKU_TEAM: iterative-sandbox
         GITHUB_MATRIX_OS: ${{ matrix.os }}
         GITHUB_MATRIX_PYTHON: ${{ matrix.python }}
+        BITBUCKET_USERNAME: ${{ secrets.BITBUCKET_USERNAME }}
+        BITBUCKET_PASSWORD: ${{ secrets.BITBUCKET_PASSWORD }}
     - name: "Upload coverage to Codecov"
       uses: codecov/codecov-action@v1
       with:

.pre-commit-config.yaml

@@ -17,8 +17,8 @@ repos:
       - id: mixed-line-ending
       - id: sort-simple-yaml
       - id: trailing-whitespace
-  - repo: 'https://gitlab.com/pycqa/flake8'
-    rev: 3.9.2
+  - repo: 'https://github.com/pycqa/flake8'
+    rev: 4.0.1
     hooks:
       - id: flake8
         args:
@@ -46,6 +46,7 @@ repos:
           - types-PyYAML
           - pydantic
           - types-filelock
+          - types-emoji
   - repo: local
     hooks:
       - id: pylint

README.md

@@ -4,10 +4,11 @@
 [![Check, test and release](https://github.com/iterative/mlem/actions/workflows/check-test-release.yml/badge.svg)](https://github.com/iterative/mlem/actions/workflows/check-test-release.yml)
 [![codecov](https://codecov.io/gh/iterative/mlem/branch/main/graph/badge.svg?token=WHU4OAB6O2)](https://codecov.io/gh/iterative/mlem)
 [![PyPi](https://img.shields.io/pypi/v/mlem.svg?label=pip&logo=PyPI&logoColor=white)](https://pypi.org/project/mlem)
+[![License: Apache 2.0](https://img.shields.io/github/license/iterative/mlem)](https://github.com/iterative/mlem/blob/master/LICENSE)
+<!-- [![Maintainability](https://codeclimate.com/github/iterative/mlem/badges/gpa.svg)](https://codeclimate.com/github/iterative/mlem) -->
 
 MLEM helps you package and deploy machine learning models.
-It saves model metadata in a standard, human-readable format that can be used in a variety of deployment scenarios, such as real-time serving through a REST API or a batch processing task.
-MLEM lets you keep Git as the single source of truth for code and models.
+It saves ML models in a standard format that can be used in a variety of production scenarios such as real-time REST serving or batch processing.
 
 - **Run your ML models anywhere:**
   Wrap models as a Python package or Docker Image, or deploy them to Heroku (SageMaker, Kubernetes, and more platforms coming soon).
@@ -19,29 +20,262 @@ MLEM lets you keep Git as the single source of truth for code and models.
 
 - **Stick to your training workflow:**
   MLEM doesn't ask you to rewrite model training code.
-  Just add two lines around your Python code: one to import the library and one to save the model.
+  Add just two lines around your Python code: one to import the library and one to save the model.
 
 - **Developer-first experience:**
-  Use the CLI when you feel like DevOps and the API when you feel like a developer.
+  Use the CLI when you feel like DevOps, or the API if you feel like a developer.
 
 ## Why is MLEM special?
 
-The main reason to use MLEM instead of other tools is to adopt a **GitOps approach**, helping you manage model lifecycles in Git:
+The main reason to use MLEM instead of other tools is to adopt a **GitOps approach** to manage model lifecycles.
 
-- **Git as a single source of truth:** MLEM writes model metadata to a plain text file that can be versioned in a Git repo along with your code.
-- **Unify model and software deployment:** Release models using the same processes used for software updates (branching, pull requests, etc.).
-- **Reuse existing Git infrastructure:** Use familiar hosting like Github or Gitlab for model management, instead of having separate services.
+- **Git as a single source of truth:**
+  MLEM writes model metadata to a plain text file that can be versioned in Git along with code.
+  This enables GitFlow and other software engineering best practices.
 
-## Installation
+- **Unify model and software deployment:**
+  Release models using the same processes used for software updates (branching, pull requests, etc.).
+
+- **Reuse existing Git infrastructure:**
+  Use familiar hosting like Github or Gitlab for model management, instead of having separate services.
+
+- **UNIX philosophy:**
+  MLEM is a modular tool that solves one problem very well.
+  It integrates well into a larger toolset from Iterative.ai, such as [DVC](https://dvc.org/) and [CML](https://cml.dev/).
+
+## Usage
+
+This a quick walkthrough showcasing deployment and export functionality of MLEM.
+
+Please read [Get Started guide](https://mlem.ai/doc/get-started) for a full version.
+
+### Installation
 
 MLEM requires Python 3.
 
 ```console
 $ pyhon -m pip install mlem
 ```
 
-To install the pre-release version, use:
+> To install the pre-release version:
+>
+> ```console
+> $ pyhon -m pip install git+https://github.com/iterative/mlem
+> ```
 
-```console
-$ pyhon -m pip install git+https://github.com/iterative/mlem
+### Saving the model
+
+```python
+# train.py
+from mlem.api import save
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.datasets import load_iris
+
+def main():
+    data, y = load_iris(return_X_y=True, as_frame=True)
+    rf = RandomForestClassifier(
+        n_jobs=2,
+        random_state=42,
+    )
+    rf.fit(data, y)
+
+    save(
+        rf,
+        "rf",
+        sample_data=data,
+        labels=["random-forest", "classifier"],
+        description="Random Forest Classifier",
+    )
+
+if __name__ == "__main__":
+    main()
+```
+
+Check out what we have:
+
+```shell
+$ ls
+rf
+rf.mlem
+$ cat rf.mlem
+```
+<details>
+  <summary>> Click to show `cat` output</summary>
+
+```yaml
+artifacts:
+  data:
+    hash: ea4f1bf769414fdacc2075ef9de73be5
+    size: 163651
+    uri: rf
+description: Random Forest Classifier
+labels:
+- random-forest
+- classifier
+model_type:
+  methods:
+    predict:
+      args:
+      - name: data
+        type_:
+          columns:
+          - sepal length (cm)
+          - sepal width (cm)
+          - petal length (cm)
+          - petal width (cm)
+          dtypes:
+          - float64
+          - float64
+          - float64
+          - float64
+          index_cols: []
+          type: dataframe
+      name: predict
+      returns:
+        dtype: int64
+        shape:
+        - null
+        type: ndarray
+    predict_proba:
+      args:
+      - name: data
+        type_:
+          columns:
+          - sepal length (cm)
+          - sepal width (cm)
+          - petal length (cm)
+          - petal width (cm)
+          dtypes:
+          - float64
+          - float64
+          - float64
+          - float64
+          index_cols: []
+          type: dataframe
+      name: predict_proba
+      returns:
+        dtype: float64
+        shape:
+        - null
+        - 3
+        type: ndarray
+    sklearn_predict:
+      args:
+      - name: X
+        type_:
+          columns:
+          - sepal length (cm)
+          - sepal width (cm)
+          - petal length (cm)
+          - petal width (cm)
+          dtypes:
+          - float64
+          - float64
+          - float64
+          - float64
+          index_cols: []
+          type: dataframe
+      name: predict
+      returns:
+        dtype: int64
+        shape:
+        - null
+        type: ndarray
+    sklearn_predict_proba:
+      args:
+      - name: X
+        type_:
+          columns:
+          - sepal length (cm)
+          - sepal width (cm)
+          - petal length (cm)
+          - petal width (cm)
+          dtypes:
+          - float64
+          - float64
+          - float64
+          - float64
+          index_cols: []
+          type: dataframe
+      name: predict_proba
+      returns:
+        dtype: float64
+        shape:
+        - null
+        - 3
+        type: ndarray
+  type: sklearn
+object_type: model
+requirements:
+- module: sklearn
+  version: 1.0.2
+- module: pandas
+  version: 1.4.1
+- module: numpy
+  version: 1.22.3
+```
+</details>
+
+### Deploying the model
+
+If you want to follow this Quick Start, you'll need to sign up on https://heroku.com,
+create an API_KEY and populate `HEROKU_API_KEY` env var.
+
+First, create an environment to deploy your model:
+
+```shell
+$ mlem declare env heroku staging
+💾 Saving env to staging.mlem
 ```
+
+Now we can [deploy the model with `mlem deploy`](https://mlem.ai/doc/get-started/deploying)
+(you need to use different `app_name`, since it's going to be published on https://herokuapp.com):
+
+```shell
+$ mlem deployment run mydeploy -m rf -t staging -c app_name=mlem-quick-start
+⏳️ Loading deployment from .mlem/deployment/myservice.mlem
+🔗 Loading link to .mlem/env/staging.mlem
+🔗 Loading link to .mlem/model/rf.mlem
+💾 Updating deployment at .mlem/deployment/myservice.mlem
+🏛 Creating Heroku App example-mlem-get-started
+💾 Updating deployment at .mlem/deployment/myservice.mlem
+🛠 Creating docker image for heroku
+  💼 Adding model files...
+  🛠 Generating dockerfile...
+  💼 Adding sources...
+  💼 Generating requirements file...
+  🛠 Building docker image registry.heroku.com/example-mlem-get-started/web...
+  ✅  Built docker image registry.heroku.com/example-mlem-get-started/web
+  🔼 Pushed image registry.heroku.com/example-mlem-get-started/web to remote registry at host registry.heroku.com
+💾 Updating deployment at .mlem/deployment/myservice.mlem
+🛠 Releasing app my-mlem-service formation
+💾 Updating deployment at .mlem/deployment/myservice.mlem
+✅  Service example-mlem-get-started is up. You can check it out at https://mlem-quick-start.herokuapp.com/
+```
+
+### Exporting the model
+
+You could easily [export the model to a different format using `mlem build`](https://mlem.ai/doc/get-started/building):
+
+```
+$ mlem build rf docker -c server.type=fastapi -c image.name=sklearn-model
+⏳️ Loading model from rf.mlem
+🛠 Building MLEM wheel file...
+💼 Adding model files...
+🛠 Generating dockerfile...
+💼 Adding sources...
+💼 Generating requirements file...
+🛠 Building docker image sklearn-model:latest...
+✅  Built docker image sklearn-model:latest
+```
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](https://mlem.ai/doc/contributing/core)
+for more details. Thanks to all our contributors!
+
+## Copyright
+
+This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root).
+
+By submitting a pull request to this project, you agree to license your contribution under the Apache license version 2.0 to this project.

mlem/__init__.py

@@ -7,11 +7,11 @@
 import mlem.log  # noqa
 
 from . import api  # noqa
-from .config import CONFIG
+from .config import LOCAL_CONFIG
 from .ext import ExtensionLoader
 from .version import __version__
 
-if CONFIG.AUTOLOAD_EXTS:
+if LOCAL_CONFIG.AUTOLOAD_EXTS:
     ExtensionLoader.load_all()
 
-__all__ = ["api", "__version__"]
+__all__ = ["api", "__version__", "LOCAL_CONFIG"]