This repository has been archived by the owner on Sep 13, 2023. It is now read-only.
Release 0.3.0 #397
Merged
Release 0.3.0 #397
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add -c help to declare WIP * extrapolate for other commands * some field docs and little improvements * cli utils * add simple_parsing * fix tests * fix tests * lazy help * ooopsie * class and fields docstrings * reparsing cli params for nested complex objects rewrite get_field_docstring for 9000x speed fix build_model bug * fix for py37 * support lists in build_model * support lists in cli * nested options WIP * very nested options WIP * all but flat nested WIP * lil refactoring * flat nested stuff DONE * Update mlem/contrib/heroku/build.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/cli/declare.py Co-authored-by: Alexander Guschin <[email protected]> * get rid of --conf, add mlem abc to declare * fix tests * fix lazyness * fix serialization * fix tests * fix tests * Update bitbucketfs.py * Apply suggestions from code review * fix comments and disable failfast for gh actions * backport docs from mlem.ai * sort import choices * make run_cmd optional instead of bool * docs for torch import * allow --load for groups add server config into docker build * fix windows bugs * suddenly fix dockerhub requests * suddenly fix dockerhub requests Co-authored-by: Alexander Guschin <[email protected]>
Codecov ReportBase: 89.49% // Head: 87.02% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #397 +/- ##
==========================================
- Coverage 89.49% 87.02% -2.47%
==========================================
Files 81 96 +15
Lines 6413 7909 +1496
==========================================
+ Hits 5739 6883 +1144
- Misses 674 1026 +352
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
* new state POC * update docker and mock deployments * add locks * simplify deployment meta (by complexifying code) * fix tests * fix tests * fix tests * fix win tests * default env and server * fsspec manager as default * Sagemaker deployments (#366) * WIP * its alive (kinda) * it works but it's ugly * little less ugly * lil fix * fix lint * fix lint * fix tests * fix tests * fix windows bugs * fix tests * fix tests * fix for dirs deployment state * create MlemSource to choose how mlem is added to docker * test that all configs in entrypoints * better cli val error * better docker package install * finish merge * fix short tests * fix short tests * Update mlem/contrib/sagemaker/runtime.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/core/objects.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/core/objects.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/contrib/docker/base.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/contrib/docker/base.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/contrib/docker/base.py Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/contrib/heroku/meta.py Co-authored-by: Alexander Guschin <[email protected]> * Apply suggestions from code review Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/contrib/sagemaker/build.py Co-authored-by: Alexander Guschin <[email protected]> Co-authored-by: Alexander Guschin <[email protected]>
* fix tests * Sagemaker deployments (#366) * WIP * its alive (kinda) * it works but it's ugly * little less ugly * lil fix * fix lint * fix lint * fix tests * fix tests * fix windows bugs * fix tests * fix tests * test that all configs in entrypoints * fix short tests * wip kubernetes support * use APIs to deploy and get status, deletion still pending * remove get client from state * fix param * fix jinja template * working remove and status * fix client * small fixes * attempt to add tests * setup github actions for k8s tests * fix linter * use predict method of client * allow registry to be configurable by cli * change calculation of host and port according to service type * re-enable k8s test as new workflow * fix daemon access in tests * make linter happy * fix fixtures * suggested fixes and refactor * make namespace as a separate field and use enums * use watcher to figure out when resources are deleted * check minikube status before loading kubeconfig in fixture * minor suggestions * use enums for comparisons as well * create abstract class for services for host and port info * raise error when service of type clusterIP * fix build and use tag as model hash * fix echo message * hot swapping of docker image deployed * remove unnecessary f-string * skip swapping when same hash is tried to be deployed again * suggested improvements * fix lint * fix pylint * suggested improvements * fix pylint * update entrypoints * add docstrings for K8sYamlBuildArgs * add docstrings for k8s service type classes * capitalize docstrings for fields * remove service type enum * Remove new workflow for K8s * remove duplicate methods * remove version from iterative-telemetry Co-authored-by: mike0sv <[email protected]>
* Fix docstrings * comments from mlem.ai PR * ignore mocks * fix tests and update metavars * dot * feedback * no dots * add exts tests * fix field doc * fix declare list and tests * no typed links in mlem list * MLEM model and MLEM dataset * 💅🏼
# Conflicts: # mlem/cli/main.py # tests/cli/test_deployment.py # tests/cli/test_serve.py
# Conflicts: # tests/test_setup.py
* initial draft * almost * fixing things * fix bug * fix links in config get/set * mark test as long * remove newline characters, add help to mlemmixin * fix linting * fix some tests * mark test as xfail * fix issues with pandas pylint and flake8 (#427) * fix issues with pandas pylint and flake8 * fix requirements * fix req tests * remove comment * fix catboost req tests Co-authored-by: mike0sv <[email protected]> Co-authored-by: mike0sv <[email protected]> Co-authored-by: Madhur Tandon <[email protected]>
* refactoring of deployments and env - move logic to deployments, remove model field * adjust deployments cli * add rev to deploy API * fix tests and pylint * add deployment declaration and model link to state * fix lint * fix tests * fix tests * fix tests * add debug env * fix crash on missing deps * linting * fix issues with pandas pylint and flake8 * fix requirements * fix req tests * remove comment * fix docker tests * fix catboost req tests * fix issues with pandas pylint and flake8 (#427) * fix issues with pandas pylint and flake8 * fix requirements * fix req tests * remove comment * fix catboost req tests Co-authored-by: mike0sv <[email protected]> * fix k8s tests * fix k8s tests * fix windows bugs * fix sagemaker circular import * fix configs * Apply suggestions from code review Co-authored-by: Alexander Guschin <[email protected]> * Update mlem/cli/deployment.py Co-authored-by: Alexander Guschin <[email protected]> Co-authored-by: Madhur Tandon <[email protected]> Co-authored-by: Madhur Tandon <[email protected]> Co-authored-by: Alexander Guschin <[email protected]>
aguschin
reviewed
Oct 5, 2022
| cls=mlem_group("runtime", aliases=["deploy"]), | ||
| ) | ||
| app.add_typer(deployment) | ||
|
|
||
| deploy_run = Typer( |
There was a problem hiding this comment.
(Using thread to keep comments under control, this is unrelated to a specific line highlighed)
Can't build docker image with this branch:
$ mlem declare env docker --registry local --daemon.host localhost docker-builder.mlem
$ mlem build docker --file_conf docker-builder --model rf.mlem --image.name=rf-docker
not enough values to unpack (expected 2, got 1)
What's wrong? @mike0sv
There was a problem hiding this comment.
btw, it's --file_conf here, but it's --load in $ mlem deployment run --load ...
There was a problem hiding this comment.
You should use mlem build --load, not mlem build docker --file_conf. I will check the bug though
There was a problem hiding this comment.
--file_conf is in $ mlem build docker -h for me. Don't see --load there.
There was a problem hiding this comment.
Ah, got you. mlem build --load, not mlem build docker --load. Do we need --file_conf though? It can be confusing, just like it was for me (I know I need to be more careful reading details, but so do other people as well :)
There was a problem hiding this comment.
The problem was that I used env instead of builder. It works like this:
$ mlem declare builder docker docker-builder.mlem --image.name rf-docker --env.daemon.host "" --server fastapi
$ mlem build --load docker-builder.mlem --model models/rfFew items:
-
--env.daemon.hostshould be marked asrequired- runningdeclarefails without it -
--servershould be marked as required arg, sincebuildfails if it wasn't specified inbuilderdeclaration - We could improve the error I mentioned in the 1st message (
not enough values to unpack (expected 2, got 1)) and print something likeFile provided tofile_confmust have lines of form "arg=value" - (UPD) let's rename
--file_confoption to--load(mlem build docker -h)
There was a problem hiding this comment.
Looks like this wasn't fixed by your last PR @mike0sv. This is not that urgent though. @madhur-tandon, could you please TAL and fix what I've reported in the last message above?
There was a problem hiding this comment.
Again, I am able to run mlem declare builder docker lold.mlem --image.name lold without passing --env.daemon.host so it is not mandatory. Looks like a local issue again on your side. Same for --server, it is NOT required for build.
There was a problem hiding this comment.
Thank you @madhur-tandon, indeed, those were issues with my installation. Could you please fix the last 2 items? They still appear for me.
There was a problem hiding this comment.
I've created an issue so we won't forgot addressing it later #441
aguschin
reviewed
Oct 6, 2022
* Fix docstrings * comments from mlem.ai PR * ignore mocks * fix tests and update metavars * dot * feedback * no dots * add exts tests * Add docstrings to extensions Co-authored-by: Alexander Guschin <[email protected]>
* get rid of mlem dir * fix tests * fix bitbucket * fix gitlab * fix other tests * fix bb tests
aguschin
reviewed
Oct 13, 2022
|
@madhur-tandon, btw this branch have few issues I've reported - they prevent me from finishing GS for build and deploy in new docs PR. If you could fix them, that would be awesome! |
* fixing docstrings to re-generate API docs * fix newline
* add bandit to pre-commit & bump hooks * fix bandit vulnerabilities * refactor: remove key `usedforsecurity`, because not all version pylint support * fix: try for `autoescape` into `jinja2.Environment * style: apply * confidense -> confidence
* add requirements builder * add builder for virtualenv * fix linting * fix pylint again * fix entrypoints * remove upgrade deps * add ability to install in current active venv * fix pylint * add ability to create conda based envs * fix import * fix has_conda check * fix windows issues * relax array comparison test * fix typo * use conda in github actions * suggested improvements * fix python version determination * make create_virtual_env consistent * suggested improvements * add test for unix based req * pass sample data * fix test * minor improvements * use load_impl_ext for requirements builder * fix tests * add test for invalid req_type * register conda mark in pytest * add test for current and active venv * fix tests * fix test * add conda reqs as list for the builder * remove usage of context and protected access * fix entrypoints * remove pylint disable * move CondaPackageRequirement to mlem.contrib.venv * use materialize * fix tests * fix docstrings * fix tests * fix docs based test * suggested changes
aguschin
approved these changes
Oct 27, 2022
There was a problem hiding this comment.
NICE! Congrats everyone! @mike0sv @madhur-tandon 🚀 this is an awesome step in the direction of the awesomeness!
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a list of activities that should be done before release announce
First, things before release
Breaking changes - merge in this PR
serve,apply-remote,buildanddeclareOther changes - merge in this PR
--quiet--quietoption #448 @mike0svDocs
Blog post
Example-repo - PR in
example-repos-devexample-mlem-get-startedUpdate the website code animations
Second, the release itself
MLEM Release
Third, things after release
example-repos-devand updateiterative/example-mlem-get-startedMLEM: release 0.3.0 example-repos-dev#139xfailmark fromtests/cli/test_main.py::test_commands_docs_linksRemovexfailmark fromtests/cli/test_main.py::test_commands_docs_links#462