Error: package directory 'nni_node' does not exist when pip installing from GitHub

[weiji14]

Describe the issue:

Hi there, just starting to use nni and was interested in trying some features in the master branch, but couldn't get the install to work. The error message is error: package directory 'nni_node' does not exist.

Environment:

• NNI version: cbb63c5
• Training service (local|remote|pai|aml|etc): local
• Client OS: LInux
• Server OS (for remote mode only):
• Python version: 3.9
• PyTorch/TensorFlow version: 1.11
• Is conda/virtualenv/venv used?: conda/mamba
• Is running in Docker?: No

Configuration:

• Experiment config (remember to remove secrets!):
• Search space:

Log message:

• nnimanager.log:
• dispatcher.log:
• nnictl stdout and stderr:

How to reproduce it?:

Run pip install git+https://github.com/microsoft/nni.git. This produces the error:

Collecting git+https://github.com/microsoft/nni.git
  Cloning https://github.com/microsoft/nni.git to /tmp/pip-req-build-2m66e716
  Running command git clone --filter=blob:none --quiet https://github.com/microsoft/nni.git /tmp/pip-req-build-2m66e716
  Resolved https://github.com/microsoft/nni.git to commit cbb63c5bec618354a25583c0861f45d4a01d9812
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info
      writing /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/entry_points.txt
      writing requirements to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/SOURCES.txt'
      package init file 'nni/algorithms/compression/tensorflow/__init__.py' not found (or not a regular file)
      package init file 'nni/algorithms/compression/v2/__init__.py' not found (or not a regular file)
      package init file 'nni/compression/pytorch/quantization/__init__.py' not found (or not a regular file)
      error: package directory 'nni_node' does not exist
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

For reference, I'm using pip=22.0.4, setuptools=62.1.0, wheel=0.37.1.

[matluster]

NNI can't be installed in this way because nni_node needs to be pre-built. See build from source.

[weiji14]

Thanks, I managed to do git clone https://github.com/microsoft/nni.git and then pip install --verbose --editable nni/, and it seems to work. That said, I noticed lots of warnings:

Installing collected packages: nni
  Running setup.py develop for nni
    Running command python setup.py develop
    running develop
    /home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    # Building NNI manager 
    yarn install v1.22.10
    [1/5] Validating package.json...
    [2/5] Resolving packages...
    success Already up-to-date.
    Done in 0.19s.
    yarn run v1.22.10
    $ tsc
    Done in 3.05s.
    # Building web UI 
    yarn install v1.22.10
    [1/4] Resolving packages...
    [2/4] Fetching packages...
    info fsevents@2.3.2: The platform "linux" is incompatible with this module.
    info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
    info fsevents@1.2.13: The platform "linux" is incompatible with this module.
    info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
    [3/4] Linking dependencies...
    warning " > [email protected]" has unmet peer dependency "postcss@^8.1.4".
    warning " > [email protected]" has unmet peer dependency "postcss@^7.0.0 || ^8.0.1".
    warning " > [email protected]" has unmet peer dependency "postcss@>= 8".
    warning "postcss-normalize > [email protected]" has unmet peer dependency "postcss@>=8".
    warning " > [email protected]" has incorrect peer dependency "webpack@^4.44.2".
    warning " > [email protected]" has incorrect peer dependency "@typescript-eslint/eslint-plugin@^4.0.0".
    warning " > [email protected]" has incorrect peer dependency "@typescript-eslint/parser@^4.0.0".
    warning " > [email protected]" has unmet peer dependency "babel-eslint@^10.0.0".
    warning Workspaces can only be enabled in private projectnames.
    [4/4] Building fresh packages...
    success Saved lockfile.
    Done in 2.79s.
    yarn run v1.22.10
    $ node --max-old-space-size=3072 scripts/developmentBuild.js
    Creating an development build...
    Compiled successfully.

    Done in 5.86s.
    # Building JupyterLab extension 
    yarn install v1.22.10
    [1/4] Resolving packages...
    success Already up-to-date.
    Done in 0.14s.
    yarn run v1.22.10
    $ tsc && jupyter labextension build .
    Building extension in .
    An error occurred.
    ValueError: Extensions require a devDependency on @jupyterlab/builder@^3.4.0, you have a dependency on 3.0.9
    See the log file for details:  /tmp/jupyterlab-debug-1nf3i9e3.log
    error Command failed with exit code 1.
    info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    # Failed to build JupyterLab extension, skip for develop mode 
    # Traceback (most recent call last):
      File "/home/username/projectname/nni/setup_ts.py", line 188, in compile_ts
        _yarn('ts/jupyter_extension', 'build')
      File "/home/username/projectname/nni/setup_ts.py", line 267, in _yarn
        subprocess.run([str(_yarn_path), *args], cwd=path, check=True, env=_yarn_env)
      File "/home/username/miniconda3/envs/projectname/lib/python3.9/subprocess.py", line 528, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['/home/username/projectname/nni/toolchain/yarn/bin/yarn', 'build']' returned non-zero exit status 1.
     
    # Creating symlinks 
    running egg_info
    writing nni.egg-info/PKG-INFO
    writing dependency_links to nni.egg-info/dependency_links.txt
    writing entry points to nni.egg-info/entry_points.txt
    writing requirements to nni.egg-info/requires.txt
    writing top-level names to nni.egg-info/top_level.txt
    package init file 'nni/algorithms/compression/tensorflow/__init__.py' not found (or not a regular file)
    package init file 'nni/algorithms/compression/v2/__init__.py' not found (or not a regular file)
    package init file 'nni/compression/pytorch/quantization/__init__.py' not found (or not a regular file)
    reading manifest file 'nni.egg-info/SOURCES.txt'
    adding license file 'LICENSE'
    writing manifest file 'nni.egg-info/SOURCES.txt'
    running build_ext
    Creating /home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/nni.egg-link (link to .)
    Adding nni 999.dev0 to easy-install.pth file
    Installing nnictl script to /home/username/miniconda3/envs/projectname/bin

    Installed /home/username/projectname/nni
Successfully installed nni-999.dev0

In particular, setup.py based installs will be deprecated in the future. Is there a timeline for nni to move to using build and/or other standards-based tools that is compliant with PEP517? I can help open a PR for this, though I'm not familiar with the node/javascript stuff, but I can give it a shot.

[matluster]

@liuzhe-lz Any comments on this?

P.S. When I looked at pyright the other day, I noticed an interesting way, that is to publish the ts package on npm, and maintain the python wrapper in another repo. We can also try that. That might solve the problem that we need a customized python setup.py develop.

[liuzhe-lz]

I don't like setuptools either but I'm afraid Python team has not worked out a serious alternative solution yet.
PEP 517 requires backends to create .whl files, but hey, where is the API? For us 3rd-party users wheel package is for command line use only. AFAIK the only documented programmatic way to build wheel package is via setuptools.setup(). Correct me if I'm wrong. I'll be excited for that.

[weiji14]

The Python packaging tools have actually matured a bit more recently. A project I help maintain switched 2 months ago (GenericMappingTools/pygmt#1797) from using setuptools to PyPA's build to build source distributions (sdist) and binary distributions (wheels), and we haven't had any issues so far. Specifically, we switched the build command from python setup.py sdist to python -m build. See also https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary

Now I'm not sure what the situation is with building a typescript package, but as @matluster mentioned above, maybe it's possible to separate the Python and Typescript components of nni? How closely integrated is the ts package here?

[cruiseliu]

The Python packaging tools have actually matured a bit more recently. A project I help maintain switched 2 months ago (GenericMappingTools/pygmt#1797) from using setuptools to PyPA's build to build source distributions (sdist) and binary distributions (wheels), and we haven't had any issues so far. Specifically, we switched the build command from python setup.py sdist to python -m build. See also https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary

Now I'm not sure what the situation is with building a typescript package, but as @matluster mentioned above, maybe it's possible to separate the Python and Typescript components of nni? How closely integrated is the ts package here?

To my understanding build is a frontend, but we are missing the backend.

NNI is not a "static" repo that you copy source files into an archive and everything done. Instead we have Python code as build hooks. Unfortunately seems the only Python build tool that supports customizing with Python code is setuptools.

Pyright is another story. It is a nodejs project with Python files. It is easy to organize a hybrid project in that way, since the nodejs toolchain has absolutely mature support for build hooks. The problem is contributors will then need to develop the project in nodejs way. They have to setup nodejs runtime on their own and install the project with npx dev. The Python devs will not be happy about that.

[cruiseliu]

Well there is another approach.
Tough not well documented, .whl format is relatively straightforward and it's not that hard to implement a non-full-featured backend by ourselves. The NNI repo should not contain this kind of code, but maybe we can host the tool on our devs' personal repo.

[weiji14]

To my understanding build is a frontend, but we are missing the backend.

Came across this backend library called Hatch (https://hatch.pypa.io/1.5). The Jupyter folks seem to have some success using it recently to build their Javascript assets (see https://blog.jupyter.org/packaging-for-jupyter-in-2022-c7be64c38926?gi=dcb04ade667a and https://github.com/jupyterlab/hatch-jupyter-builder). Haven't tried it personally, but just thought it might be of interest.