mlstacks integration (and deprecation of old deployment logic)

[strickvl]

This PR contains the necessary changes following the update / reworking of the mlops-stacks repository. Most updates relate to the ZenML CLI logic but there has also been one update to the component & stack models so that required the database to be updated.

(Note that the zenml deploy ... CLI command is untouched (for now) but we will want to revisit that in the future.)

⦾ Core concepts / chunks

Looking at the code for the zenml stack deploy ... CLI command is probably the best way to get a sense of how the flow works, but in a nutshell:

• the CLI params / arguments passed in get converted into mlstacks Pydantic models
• These are then used to create / write multiple spec files (1 defines the stack, and then there is one more spec file for each stack component). Spec files are saved in the click config directory for mlstacks. (i.e. not zenml).
• We then call the mlstacks terraform_utils module to deploy / destroy etc as needed. (For more on that logic, visit the sister PR updating the mlops-stacks repository.)
• Then finally we use whatever outputs come from the terraform process to update / amend the ZenML stacks / components.

👷 How to try me out

First things first: you'll need to install the package into your local environment. First you'll need to clone the mlops-stacks repo and check it out to the current working branch for this change:

git clone -b feature/PLATFORM-176-stacks-goes-py-pi [email protected]:zenml-io/mlops-stacks.git

# adapt the `.` in this command so that you have mlstacks and zenml both installed in the same environment
pip install -e ".[dev]" # install mlstacks

There are two core sets of functionality that mlstacks enables through ZenML:

1. stack deployment (made up of several components)
2. individual component deployment

(There is some remaining functionality from the previous logic that I left in (zenml stack recipe ... subcommands for info and describe and version) but perhaps it's better to remove those as well.)

Both use cases allow for deploy and destroy. When you run zenml stack/component describe you also get a link to your stack / component spec file printed out. This can be much improved but for this PR probably not enough time / out of scope.

🥞 Stack Deployment Example Use

To deploy a stack that is made up of just an artifact store (quickest deployment of all our options) to GCP:

zenml stack deploy -p gcp -a -n basic -r us-east1 -t z-env=dev -t z-owner=YOUR_NAME -t z-project=mlstacks -t z-team=platform -x bucket_name=zenml-meets-mlstacks -x project_id=zenml-core

This will import your stack to ZenML by default as well.

You can run it with the -b flag to enable debug mode (which basically drops you more into the terraform weeds, should you want).

To see some information about the stack you just registered:

zenml stack describe basic

To destroy the stack + its components recursively (i.e. just the artifact store) + (optionally) delete/deregister it from ZenML:

zenml stack destroy -p gcp basic

Obviously you can / should play around with different providers / components / component combinations.

🧲 Component Deployment

To deploy an artifact store component (quickest deployment of all our options) to GCP:

zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml-core new_one

This will import your component to ZenML by default as well, but then you have to decide what you want to do with it in terms of associating it with a particular stack e.g..

You can run it with the -d flag to enable debug mode (which basically drops you more into the Terraform weeds, should you want).

To see some information about the component you just registered:

zenml artifact-store describe new_one

To destroy the artifact store + (optionally) delete/deregister it from ZenML:

zenml artifact-store destroy new_one -p gcp

Obviously you can / should play around with different providers / components / component combinations.

🚧 Remaining / missing tasks

• Documentation updates on the ZenML documentation side (also required for a standalone mlstacks site but that'll happen elsewhere)
• Add analytics back in
• Adding deprecated commands back in (I just removed them while working, but they should actually just give an error message.)
• Add unit tests for new mlstacks util module
• Address mypy / listing / CI failures
• Update examples (?)
• Incorporate Terraform removal PR

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• If I have added an integration, I have updated the integrations table and the corresponding website section.
• I have added tests to cover my changes.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)