…#1811)

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* fix tests in docker

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

---------

Co-authored-by: Felix Altenberger <[email protected]>

* Delete dead code

* Implement zenml.log_artifact_metadata()

* Improve error handling and add tests

* Remove 'description' arg of log_artifact_metadata() and fix docstring

* Fix test name

* Remove TODOs

* Rewrite create_run_metadata() to make batch API requests

* Adjust unit tests

* Fix flaky test_is_secret_reference

* Fix integration tests

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* wip

* wip

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* ModelConfig implementation

* start testing

* fix tests in docker

* more tests

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

* lint

* remove `ModelStages._members()`

* improve wording

* Union[str, ModelStages]

* simplify `get_model_version` to one endpoint

* remove `_get_request_params`

* split `_get_or_create_model_version`

* ensure `get_or_create_model` is stable

* add docstrings

* lint

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

---------

Co-authored-by: Felix Altenberger <[email protected]>

…1819)

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* wip

* wip

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* ModelConfig implementation

* start testing

* fix tests in docker

* more tests

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

* add ModelConfig to step deco

* Revert "add ModelConfig to step deco"

This reverts commit ba9fd6a.

* add ModelConfig to step deco

* lint

* wip

* Add model_config to @pipeline and @step

* add docstring

* cleaning up after merge

* move `ModelBaseModel` back to `models`

* direct imports

* breaking another circular dep

* and one more circular dep

---------

Co-authored-by: Felix Altenberger <[email protected]>

…agement in context (partially) (#1822)

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* wip

* wip

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* ModelConfig implementation

* start testing

* fix tests in docker

* more tests

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

* add ModelConfig to step deco

* Revert "add ModelConfig to step deco"

This reverts commit ba9fd6a.

* add ModelConfig to step deco

* lint

* wip

* Add model_config to @pipeline and @step

* add artifact config and necessary registrations around it

* fix bug with `Output` annotation

* add docstring

* Update src/zenml/new/pipelines/pipeline.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* cleaning up after merge

* move `ModelBaseModel` back to `models`

* direct imports

* breaking another circular dep

* clean-up after conflicts resolution

* and one more circular dep

* and one more circular dep

* update tests to include `OutputSignature`

* add `link_output_to_model`

* stabilize tests

* add implicit linkage

* more tests

* lint

* docstrings

* test/bug fixes

* extend artifact link key to pipe/step/name

* lint

* fix for 3.8

* fix failing test call signatures

* update_forward_refs centrally

* Update src/zenml/model/artifact_config.py

Co-authored-by: Felix Altenberger <[email protected]>

* Update src/zenml/zen_stores/sql_zen_store.py

Co-authored-by: Felix Altenberger <[email protected]>

* add a warning about not supported feature

* add arguments descriptions

* improve docstrings a bit

* assign_version_to_running internal + docs

* rename to link_version

* Update src/zenml/new/pipelines/pipeline.py

Co-authored-by: Felix Altenberger <[email protected]>

* rename arg in test

* lint

* refactor a bit

* address PR comment

* remove deleted classes

* Update src/zenml/models/model_base_model.py

Co-authored-by: Felix Altenberger <[email protected]>

* move `ModelStages` to enums

* merge `model_stage` and add tests for it

* simplify `_link_artifacts_to_model` args

* docstrings

* docstrings

* lint

* fix args in tests

* fix __mro__ test failure

* fix for mlflow test

* stabilize test_list_runs_is_ordered

* fix pagination in some tests

* address PR comments

---------

Co-authored-by: Felix Altenberger <[email protected]>
Co-authored-by: Alex Strick van Linschoten <[email protected]>

…delete_new_version_on_failure ` option (#1825)

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* wip

* wip

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* ModelConfig implementation

* start testing

* fix tests in docker

* more tests

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

* add ModelConfig to step deco

* Revert "add ModelConfig to step deco"

This reverts commit ba9fd6a.

* add ModelConfig to step deco

* lint

* wip

* Add model_config to @pipeline and @step

* add artifact config and necessary registrations around it

* fix bug with `Output` annotation

* add docstring

* delete running versions on fail

* add deletion test

* don't return running without recovery

* remove base_model.py

* PR comments

* remove merged migration

* PR comments

* improve model_config warnings

* clean up merge mess

* bandit is too strict

* typos

* clean up merge mess

* remove not relevant asserts

* improve docs

* improve docs

* rely on deployment in `get_new_version_requests`

* stabilize tests in random order

* improve docs

* stabilize tests

---------

Co-authored-by: Felix Altenberger <[email protected]>

* big bang commit

* typo

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* add Alembic

* lint

* mypy

* darglint

* wip

* wip

* wip

* wip

* add endpoints

* add ModelStages

* wip

* work with client

* handle tags

* fix integrations

* move list around

* update db schema

* wip

* lint

* sync with model branch

* wip

* refactor

* add stage transition

* add update interface

* add model version links

* lint

* fix crud tests

* fix alembic branching

* patch azure

* lint

* use zenml StrEnum

* fix param name

* ModelConfig implementation

* start testing

* fix tests in docker

* more tests

* fix tests for mysql

* rename artifact ids variables

* reorder methods

* add direct getters

* lint

* split links into 2 tables

* lint

* pr comments

* add ModelConfig to step deco

* Revert "add ModelConfig to step deco"

This reverts commit ba9fd6a.

* add ModelConfig to step deco

* lint

* wip

* Add model_config to @pipeline and @step

* add artifact config and necessary registrations around it

* fix bug with `Output` annotation

* add docstring

* delete running versions on fail

* add deletion test

* Update src/zenml/new/pipelines/pipeline.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* cleaning up after merge

* move `ModelBaseModel` back to `models`

* direct imports

* breaking another circular dep

* clean-up after conflicts resolution

* and one more circular dep

* and one more circular dep

* update tests to include `OutputSignature`

* add `link_output_to_model`

* stabilize tests

* add implicit linkage

* more tests

* lint

* docstrings

* test/bug fixes

* extend artifact link key to pipe/step/name

* lint

* fix for 3.8

* fix failing test call signatures

* update_forward_refs centrally

* Update src/zenml/model/artifact_config.py

Co-authored-by: Felix Altenberger <[email protected]>

* Update src/zenml/zen_stores/sql_zen_store.py

Co-authored-by: Felix Altenberger <[email protected]>

* add a warning about not supported feature

* add arguments descriptions

* improve docstrings a bit

* assign_version_to_running internal + docs

* rename to link_version

* Update src/zenml/new/pipelines/pipeline.py

Co-authored-by: Felix Altenberger <[email protected]>

* rename arg in test

* lint

* refactor a bit

* address PR comment

* remove deleted classes

* Update src/zenml/models/model_base_model.py

Co-authored-by: Felix Altenberger <[email protected]>

* move `ModelStages` to enums

* merge `model_stage` and add tests for it

* simplify `_link_artifacts_to_model` args

* docstrings

* docstrings

* lint

* fix args in tests

* fix __mro__ test failure

* fix for mlflow test

* stabilize test_list_runs_is_ordered

* update .gitignore

* fix pagination in some tests

* don't return running without recovery

* remove base_model.py

* PR comments

* remove merged migration

* PR comments

* improve model_config warnings

* clean up merge mess

* external artifacts for model watchtower

* bandit is too strict

* typos

* clean up merge mess

* remove not relevant asserts

* improve docs

* improve docs

* rely on deployment in `get_new_version_requests`

* stabilize tests in random order

* improve docs

* use utils model_killer

* clean up merge mess

* use Annotated from typing_extensions

* one more circular dep case

* lint

* resolve alembic branching

* Apply suggestions from code review

Co-authored-by: Felix Altenberger <[email protected]>

* get rid of `_testable_upload_if_necessary`

* refactor `upload_if_necessary`

* `zenml.artifacts.external_artifact.ExternalArtifact`

* add tests for getters of MV response

* improve docstring

* fix before  #1835

* lint

* remove  `_import_client`

* fixing issues highlighted by tests

* update docstrings

* fix year in license

* add docstrings

* Update src/zenml/models/model_models.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/models/model_models.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/models/model_models.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/models/model_models.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/steps/external_artifact.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* Update src/zenml/models/model_models.py

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* fix docstrings

* Move UnmaterializedArtifact to zenml.artifacts

* add fix of path in docs

* split ExternalArtifact into user class and config for step

* update template working branch

* restrict pydantic

* Revert "restrict pydantic"

This reverts commit b94e01f.

---------

Co-authored-by: Felix Altenberger <[email protected]>
Co-authored-by: Alex Strick van Linschoten <[email protected]>

* extend client for Model WatchTower

* resolve alembic branching

* linkage of pipeline runs

* add docstring

* tricky bug

* full linkage on consumption

* lint

* use client

* Auto-update of E2E template

* update outdated test

* create new version if any step touching it is executed and it was requested elsewhere

* refactor `_link_pipeline_run_to_model`

* add few more tests

---------

Co-authored-by: GitHub Actions <[email protected]>

…ature/OSS-2300-model-watch-tower-v0.1

* linkage of pipeline runs

* add docstring

* tricky bug

* full linkage on consumption

* lint

* use client

* Auto-update of E2E template

* update outdated test

* create new version if any step touching it is executed and it was requested elsewhere

* refactor `_link_pipeline_run_to_model`

* add few more tests

* parallel execution of model versions

* add version number

* improve readability

* protect from misuse

* extend `ArtifactConfig.model_version`

* align model config docstrings

* stabilize parallelized test

* rework test as subprocess calls

* skip subprocess test on windows

* after merge mess

* update tests flow based on develop

* proper handle __latest__ mv in REST

* fix get model version endpoint

* simplify user-facing interface

* fix test annotation

---------

Co-authored-by: GitHub Actions <[email protected]>

* linkage of pipeline runs

* add docstring

* tricky bug

* full linkage on consumption

* lint

* use client

* Auto-update of E2E template

* update outdated test

* create new version if any step touching it is executed and it was requested elsewhere

* refactor `_link_pipeline_run_to_model`

* add few more tests

* parallel execution of model versions

* add version number

* improve readability

* protect from misuse

* extend `ArtifactConfig.model_version`

* align model config docstrings

* stabilize parallelized test

* rename `model.py`

* rename

* add cli for model watchtower

* rework test as subprocess calls

* Apply suggestions from code review

Co-authored-by: Alex Strick van Linschoten <[email protected]>

* reshape tests

* rebranding

* skip subprocess test on windows

* sort init

* remove redundant param from `print_pydantic_models`

* reduce redundant code

* reduce too long line

* add model update

* after merge mess

* lint

* update tests flow based on develop

* proper handle __latest__ mv in REST

* fix get model version endpoint

* calm down linter

* restore proper name

* typing import error

* after merge mess

---------

Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Alex Strick van Linschoten <[email protected]>

…hub.com/zenml-io/zenml into feature/OSS-2300-model-watch-tower-v0.1

…hub.com/zenml-io/zenml into feature/OSS-2300-model-watch-tower-v0.1