[MO] pip packaging #3123

dkurt · 2020-11-13T10:59:33Z

Usage

import mo_tf
import sys
import subprocess

subprocess.run([sys.executable,
                mo_tf.__file__,
                '--input_model', 'mobilenet_v1_1.0_224_frozen.pb',
                '--input_shape', '[1,224,224,3]'],
               check=True)

Install

now

pip install git+https://github.com/dkurt/openvino.git@mo_pip#subdirectory=model-optimizer

after merge

pip install git+https://github.com/openvinotoolkit/openvino.git#subdirectory=model-optimizer

or by wheel

Key features:

No test data (*_test.py and test-generator)
Only minimal set of dependencies (no frameworks) to make package light

dkurt · 2020-12-30T10:11:26Z

@lazarevevgeny, @mryzhov, please review updated version of PR

model-optimizer/setup.py

SDxKeeper · 2021-01-12T09:56:20Z

@dkurt Can you please describe how user should use this package?

Currently package installs requirements_*.txt files directly into root of virtual env if user uses it. Will you add documentation to the package how to install deps?
Cloning full openvino repo is quite long process. Will there be a pip distribution of release on pypi.org similar to https://pypi.org/project/openvino/ package
Using subprocess is good for first step, do you see direct MO software API usage from python code in the future (without spawning new process)?

dkurt · 2021-01-12T10:25:03Z

@dkurt Can you please describe how user should use this package?

See example here: deepchem/deepchem#2332. We need this usage to use MO from GitHub as pip dependency.

Currently package installs requirements_*.txt files directly into root of virtual env if user uses it. Will you add documentation to the package how to install deps?

Not clear for me what kind of documentations should be added? requirements_*.txt is required by MO and that's why it's added to the package.

Cloning full openvino repo is quite long process. Will there be a pip distribution of release on pypi.org similar to https://pypi.org/project/openvino/ package

setup.py still required for both. You may install in two ways: as git repo or as a wheel. See the tests workflow with wheel example.

Using subprocess is good for first step, do you see direct MO software API usage from python code in the future (without spawning new process)?

There were a proposal at initial commit 6a663f3. But we need this feature ASAP so don't want to block on API review.

model-optimizer/setup.py

lazarevevgeny · 2021-01-13T08:47:38Z

model-optimizer/setup.py

+    def find_package_modules(self, package, package_dir):
+        modules = super().find_package_modules(package, package_dir)
+        return [
+            (pkg, module, filename)
+            for (pkg, module, filename) in modules
+            if not filename.endswith('_test.py')
+        ]


Can we reuse the CMake script which is already available for the MO to get list of files which needes to be included into the package?

CC @slyubimt

Not sure that CMake dependency is a good option.

There is a script "/automation/create_package.py" which will create an archive with all necessary files. Instead of implementing logic to determine list of files to be included in the wheel we can re-use this script.

model-optimizer/setup.py

lazarevevgeny · 2021-01-13T08:49:20Z

model-optimizer/setup.py

+        path = os.path.join(self.install_purelib, package_name, '__init__.py')
+        with open(path, 'wt') as f:
+            f.write('import os, sys\n')
+            f.write('from {} import mo\n'.format(package_name))
+            # This is required to fix internal imports
+            f.write('sys.path.append(os.path.dirname(__file__))\n')
+            # We install a package into custom folder "package_name".
+            # Redirect import to model-optimizer/mo/__init__.py
+            f.write('sys.modules["mo"] = mo')


Can we change something in the MO so we could get rid of this workaround?

dkurt · 2021-01-13T09:32:18Z

@SDxKeeper,

Currently package installs requirements_*.txt files directly into root of virtual env if user uses it. Will you add documentation to the package how to install deps?

I got your point, sorry for confusion. I'll try to fix that.

UPD: Solved

model-optimizer/setup.py

SDxKeeper · 2021-01-19T14:11:19Z

Currently package installs requirements_*.txt files directly into root of virtual env if user uses it. Will you add documentation to the package how to install deps?

Not clear for me what kind of documentations should be added? requirements_*.txt is required by MO and that's why it's added to the package.

What I mean here is that wrapper which you are making at https://github.com/deepchem/deepchem/pull/2332/files#diff-26fde86013198b5c8d4604dae3d3bd4ed19c9aa18de4db58a4b83dd786c6613fR128 calls MO as it's already installed and configured for tf usage, but MO requires installation of requirements_tf.txt or requirements_tf2.txt before it can be used.
How will user find out that he needs to go to venv folder and call installation of venv/requirements_*.txt files from unusual place in deepchem scenario?

dkurt · 2021-01-21T10:53:17Z

@SDxKeeper, I fixed this problem so no venv/requirements_*.txt is installed.

SDxKeeper · 2021-01-22T14:30:37Z

@dkurt thank you for removing venv/requirements*.txt files. Sorry I didn't notice update on comment 9 days ago.

mryzhov

@dkurt could you please rebase PR to rerun the tests

clouds56 · 2021-08-15T02:20:32Z

Cloning full openvino repo is quite long process. Will there be a pip distribution of release on pypi.org similar to https://pypi.org/project/openvino/ package

setup.py still required for both. You may install in two ways: as git repo or as a wheel. See the tests workflow with wheel example.

Shall we release a wheel (maybe named openvino-mo) on pypi.org?

dkurt · 2021-08-15T09:58:07Z

@clouds56, there is already openvino-dev. Does it fit your requirements?

clouds56 · 2021-08-16T03:15:34Z

@dkurt thanks for the information! That's exactly what I need.
(Maybe we should add a link to openvino-dev in package openvino)

dkurt requested a review from a team November 13, 2020 10:59

dkurt marked this pull request as draft November 13, 2020 10:59

openvino-pushbot added the category: MO Model Optimizer label Nov 13, 2020

dkurt force-pushed the mo_pip branch from 597a363 to 563f3fb Compare November 16, 2020 12:59

dkurt marked this pull request as ready for review November 16, 2020 13:13

dkurt force-pushed the mo_pip branch from 563f3fb to 86dc23e Compare November 16, 2020 13:14

dkurt requested review from mryzhov and lazarevevgeny November 16, 2020 13:15

dkurt mentioned this pull request Dec 17, 2020

Enabled wheel packaging #3582

Merged

dkurt requested a review from a team December 30, 2020 08:43

dkurt force-pushed the mo_pip branch 8 times, most recently from 976928d to a68e5ad Compare December 30, 2020 10:02

dkurt force-pushed the mo_pip branch from 2ac8abb to 615bf05 Compare January 11, 2021 08:39

dkurt mentioned this pull request Jan 11, 2021

Intel OpenVINO backend deepchem/deepchem#2332

Closed

14 tasks

azhogov requested a review from generalova-kate January 12, 2021 06:24

generalova-kate reviewed Jan 12, 2021

View reviewed changes

model-optimizer/setup.py Outdated Show resolved Hide resolved

dkurt force-pushed the mo_pip branch from 615bf05 to de445e1 Compare January 12, 2021 07:27

SDxKeeper reviewed Jan 12, 2021

View reviewed changes

model-optimizer/setup.py Outdated Show resolved Hide resolved

lazarevevgeny suggested changes Jan 13, 2021

View reviewed changes

lazarevevgeny requested a review from slyalin January 13, 2021 08:49

dkurt force-pushed the mo_pip branch from f2d2cbd to 4cc1c4e Compare January 13, 2021 09:03

dkurt force-pushed the mo_pip branch from 4cc1c4e to 1470fa7 Compare January 13, 2021 09:17

mryzhov reviewed Jan 14, 2021

View reviewed changes

model-optimizer/setup.py Show resolved Hide resolved

ilya-lavrenov requested a review from slyubimt January 15, 2021 21:50

mryzhov assigned ilya-lavrenov, ilyachur and lazarevevgeny Feb 8, 2021

mryzhov added this to the 2021.3 milestone Feb 8, 2021

dkurt added 6 commits February 8, 2021 14:30

[MO] pip packaging

06d5a5a

Use subprocess

eab54cf

Remove MANIFEST.in

9895660

Fix comments

6b344e5

Fix requirements.txt duplication

ad523bb

Add license

2526e27

mryzhov approved these changes Feb 8, 2021

View reviewed changes

dkurt force-pushed the mo_pip branch from 5efd8bb to 2526e27 Compare February 8, 2021 11:36

ilya-lavrenov removed their assignment Feb 8, 2021

lazarevevgeny approved these changes Feb 8, 2021

View reviewed changes

mryzhov merged commit 14d3310 into openvinotoolkit:master Feb 9, 2021

dkurt deleted the mo_pip branch February 18, 2021 14:18

dkurt mentioned this pull request Feb 25, 2021

cgcnn dkurt/deepchem#7

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MO] pip packaging #3123

[MO] pip packaging #3123

dkurt commented Nov 13, 2020 •

edited

Loading

dkurt commented Dec 30, 2020

SDxKeeper commented Jan 12, 2021 •

edited

Loading

dkurt commented Jan 12, 2021 •

edited

Loading

lazarevevgeny Jan 13, 2021

dkurt Jan 13, 2021

lazarevevgeny Jan 15, 2021

lazarevevgeny Jan 13, 2021

dkurt commented Jan 13, 2021 •

edited

Loading

SDxKeeper commented Jan 19, 2021

dkurt commented Jan 21, 2021

SDxKeeper commented Jan 22, 2021

mryzhov left a comment

clouds56 commented Aug 15, 2021

dkurt commented Aug 15, 2021

clouds56 commented Aug 16, 2021

[MO] pip packaging #3123

[MO] pip packaging #3123

Conversation

dkurt commented Nov 13, 2020 • edited Loading

Usage

Install

Key features:

dkurt commented Dec 30, 2020

SDxKeeper commented Jan 12, 2021 • edited Loading

dkurt commented Jan 12, 2021 • edited Loading

lazarevevgeny Jan 13, 2021

Choose a reason for hiding this comment

dkurt Jan 13, 2021

Choose a reason for hiding this comment

lazarevevgeny Jan 15, 2021

Choose a reason for hiding this comment

lazarevevgeny Jan 13, 2021

Choose a reason for hiding this comment

dkurt commented Jan 13, 2021 • edited Loading

SDxKeeper commented Jan 19, 2021

dkurt commented Jan 21, 2021

SDxKeeper commented Jan 22, 2021

mryzhov left a comment

Choose a reason for hiding this comment

clouds56 commented Aug 15, 2021

dkurt commented Aug 15, 2021

clouds56 commented Aug 16, 2021

dkurt commented Nov 13, 2020 •

edited

Loading

SDxKeeper commented Jan 12, 2021 •

edited

Loading

dkurt commented Jan 12, 2021 •

edited

Loading

dkurt commented Jan 13, 2021 •

edited

Loading