Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standard wheel packages #34450

Closed
mkoeppe opened this issue Aug 29, 2022 · 34 comments
Closed

Standard wheel packages #34450

mkoeppe opened this issue Aug 29, 2022 · 34 comments

Comments

@mkoeppe
Copy link
Contributor

mkoeppe commented Aug 29, 2022

We add a mechanism to install Python packages from (platform-independent) wheels instead of from source distributions.

This will allow us to make packages such as furo and jupyterlab standard. Currently they are pip packages, as they need internet connectivity to get Javascript packages (via nodeenv).

Here we make furo a standard wheel package. This is for #33833.

Depends on #34421

CC: @dimpase @kwankyu @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch/Commit: d887d44

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/34450

@mkoeppe mkoeppe added this to the sage-9.7 milestone Aug 29, 2022
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

Author: Matthias Koeppe

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

Branch: u/mkoeppe/standard_wheel_packages

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

Commit: fd1c677

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

New commits:

8d90b56build/pkgs/furo: Make it a normal package
28f30acbuild/bin/sage-spkg: Handle *.whl packages
fd1c677build/pkgs/furo: Use a wheel

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Changed commit from fd1c677 to 9fa490a

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

9fa490abuild/pkgs/sphinx_basic_ng: New (furo dependency)

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

eba7f6fsrc/doc/en/developer/packaging.rst: Document wheel packages

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Changed commit from 9fa490a to eba7f6f

@mkoeppe

This comment has been minimized.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

8e06d43src/doc/en/developer/packaging.rst: Policy on wheels, on standard packages

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Changed commit from eba7f6f to 8e06d43

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:8

If non-platform is not already a standard term, then I suggest platform-independent, which is longer but easier to understand and reads well than non-platform

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

comment:9

Please feel free to push improvements to the documentation to the ticket. I have to stop here for tonight.

I don't see "platform-independent" in https://peps.python.org/pep-0425/; it uses the terminology "pure Python". There might be other places to look at

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:10

This failure seems related:

Installing jupyter_jsmol-2022.1.0
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Looking in links: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels
Processing /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Looking in links: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels
  ERROR: Could not find a version that satisfies the requirement jupyter_packaging~=0.7.9 (from versions: none)
  ERROR: No matching distribution found for jupyter_packaging~=0.7.9
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin/python3 /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-standalone-pip-lw2mb0lw/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-l0gj_p8c/overlay --no-warn-script-location --no-binary :all: --only-binary :none: --no-index --find-links /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels -- 'jupyter_packaging~=0.7.9' 'jupyterlab>=3.0.0,==3.*' 'setuptools>=40.8.0' wheel
  cwd: [inherit]
  Installing build dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Warning: building with "python3 -m pip wheel --wheel-dir=dist --verbose --no-deps --no-index --isolated --ignore-requires-python --find-links=/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels" failed.
Retrying with "python3 -m pip wheel --wheel-dir=dist --verbose --no-deps --no-index --isolated --ignore-requires-python --no-build-isolation --no-binary :all:".
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Processing /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  setup.py entered
  $PATH=/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/src/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/usr/local/opt/polymake/bin:/usr/local/opt/texinfo/bin:/usr/local/opt/gettext/bin:/Applications/google-cloud-sdk/bin:.:/Users/kwankyu/Workplace/bin:/usr/local/bin:/usr/local/sbin:/usr/texbin:/Applications/Macaulay2-1.20/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Magma
  <string>:47: DeprecatedWarning: create_cmdclass is deprecated as of 0.8 and will be removed in 1.0. "
  Use `wrap_installers` to handle prebuild steps in cmdclass.
  Use `get_data_files` to handle data files.
  Use `include_package_data=True` and `MANIFEST.in` for package data.

  <string>:49: DeprecatedWarning: install_npm is deprecated as of 0.8 and will be removed in 1.0. Use `npm_builder` and `wrap_installers`
  running dist_info
  creating /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info
  writing /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/dependency_links.txt
  writing requirements to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/requires.txt
  writing top-level names to /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/top_level.txt
  writing manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'examples'
  no previously-included directories found matching 'tests/build'
  no previously-included directories found matching '**/node_modules'
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
  warning: no previously-included files matching '*.map' found anywhere in distribution
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol.egg-info/SOURCES.txt'
  creating '/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-modern-metadata-_nd6zlfl/jupyter_jsmol-2022.1.0.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: jupyter-jsmol
  WARNING: Ignoring --build-option when building jupyter-jsmol using PEP 517
  Building wheel for jupyter-jsmol (pyproject.toml): started
  Running command Building wheel for jupyter-jsmol (pyproject.toml)
  setup.py entered
  $PATH=/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin:/Users/kwankyu/GitHub/sage-temp/build/bin:/Users/kwankyu/GitHub/sage-temp/src/bin:/Users/kwankyu/GitHub/sage-temp/local/bin:/usr/local/opt/polymake/bin:/usr/local/opt/texinfo/bin:/usr/local/opt/gettext/bin:/Applications/google-cloud-sdk/bin:.:/Users/kwankyu/Workplace/bin:/usr/local/bin:/usr/local/sbin:/usr/texbin:/Applications/Macaulay2-1.20/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Magma
  <string>:47: DeprecatedWarning: create_cmdclass is deprecated as of 0.8 and will be removed in 1.0. "
  Use `wrap_installers` to handle prebuild steps in cmdclass.
  Use `get_data_files` to handle data files.
  Use `include_package_data=True` and `MANIFEST.in` for package data.

  <string>:49: DeprecatedWarning: install_npm is deprecated as of 0.8 and will be removed in 1.0. Use `npm_builder` and `wrap_installers`
  running bdist_wheel
  running jsdeps
  Installing build dependencies with npm.  This may take a while...
  > yarn install
  yarn install v1.22.19
  warning package.json: No license field
  warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
  warning [email protected]: No license field
  [1/4] Resolving packages...
  [2/4] Fetching packages...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  info There appears to be trouble with your network connection. Retrying...
  error An unexpected error occurred: "https://registry.yarnpkg.com/@jupyterlab/nbformat/-/nbformat-3.1.18.tgz: tunneling socket could not be established, cause=connect ETIMEDOUT 192.0.2.0:5187".
  info If you think this is a bug, please open a bug report with the information provided in "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src/js/yarn-error.log".
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  Traceback (most recent call last):
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 244, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
      self.run_setup()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(code, locals())
    File "<string>", line 95, in <module>
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 177, in setup
      return run_commands(dist)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 193, in run_commands
      dist.run_commands()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
      self.run_command(cmd)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
      cmd_obj.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 725, in run
      [self.run_command(cmd) for cmd in cmds]
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 725, in <listcomp>
      [self.run_command(cmd) for cmd in cmds]
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 317, in run_command
      self.distribution.run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
      cmd_obj.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 344, in run
      c.run()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 694, in run
      builder()
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 230, in builder
      run(npm_cmd + ["install"], cwd=node_package)
    File "/Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_packaging/setupbase.py", line 296, in run
      return subprocess.check_call(cmd, **kwargs)
    File "/usr/local/Cellar/[email protected]/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/yarn', 'install']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × Building wheel for jupyter-jsmol (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/bin/python3 /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/tmp1q2r7ekk
  cwd: /Users/kwankyu/GitHub/sage-temp/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/jupyter_jsmol-2022.1.0/src
  Building wheel for jupyter-jsmol (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for jupyter-jsmol
Failed to build jupyter-jsmol
ERROR: Failed to build one or more wheels
********************************************************************************
Error building a wheel for jupyter_jsmol-2022.1.0
******************************************************************************** 

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:12

Replying to @mkoeppe:

I don't see "platform-independent" in https://peps.python.org/pep-0425/; it uses the terminology "pure Python". There might be other places to look at

Just from internet search, I see no "non-platform" and a couple of "platform independent". As far as I understand, "pure Python" is not quite right.


New commits:

b12e9dcUse the name platform-independent

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

Changed commit from 8e06d43 to b12e9dc

@dimpase
Copy link
Member

dimpase commented Aug 29, 2022

comment:13

You need #34421, where this jsmol issue was fixed. (Volker just merged it, but it's not yet in a beta)


New commits:

b12e9dcUse the name platform-independent

@dimpase
Copy link
Member

dimpase commented Aug 29, 2022

Dependencies: #34421

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:14

With #34421 and this ticket, #33833 works well!

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Changed commit from b12e9dc to d887d44

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 29, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

d887d44Use rerun instead of re-run

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:16

LGTM.

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

Reviewer: Kwankyu Lee

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

comment:17

Thanks!

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:19

Do you know why the github actions failed?

@kwankyu

This comment has been minimized.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

comment:20
[sphinx_basic_ng-0.0.1a12] cat: /sage/build/pkgs/sphinx_basic_ng/type: No such file or directory

Maybe I forgot to check in a file?

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:21

The file is there.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

comment:22

It's a defect of the type of incremental build with a changed SAGE_ROOT, specific to the GH Actions workflow. Nothing wrong with this ticket.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 29, 2022

comment:23

I've opened #34452 for it

@kwankyu
Copy link
Collaborator

kwankyu commented Aug 29, 2022

comment:24

Replying to @mkoeppe:

I've opened #34452 for it

Thanks.

Sage development is like mathematics research: if you solved a problem, you see a new problem from it. It never ends :)

@vbraun
Copy link
Member

vbraun commented Sep 7, 2022

Changed branch from u/klee/standard_wheel_packages to d887d44

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants