Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert remaining python2 applications over to python3 #101964

Closed
76 of 77 tasks
jonringer opened this issue Oct 28, 2020 · 54 comments · Fixed by #104726
Closed
76 of 77 tasks

Convert remaining python2 applications over to python3 #101964

jonringer opened this issue Oct 28, 2020 · 54 comments · Fixed by #104726

Comments

@jonringer
Copy link
Contributor

jonringer commented Oct 28, 2020

In #101929 we have many important programs (e.g. cachix) still using python2 in their builds. This list doesn't constitute python2 applications, but rather packages which have python2Packages.cryptography somewhere in their dependency graph. So these will be affected when python2Packages.cryptography does get marked as vulnerable.

This issue to track the conversion process over to python3, packages still needing to be converted are listed below. This list isn't exhaustive, just those that use the soon-to-be-marked-vulnerable pythonPackages.cryptography:

Finding the dependency

For most dependencies, it should be pretty obvious where python2 comes from, for more "difficult" packages. You may need to do some digging.

nix-tree + nix-instantiate

you can run nix-shell -p nix-tree --run "nix-tree $(nix-instantiate default.nix -A <package>) to get the entire build dependency tree, then search for the cryptography package, and then you should be able to trace which dependencies are introducing it.

nix why-depends

alternatively, you can use nix why-depends
nix why-depends --all -f default.nix <package> python2Packages.cryptography can also be used, however, this will require you to re-build the package, which may take more time than parsing the dependency tree above

@jonringer jonringer added the 0.kind: bug Something is broken label Oct 28, 2020
@sikmir sikmir mentioned this issue Oct 28, 2020
10 tasks
@jonringer jonringer added 0.kind: enhancement Add something new 3.skill: good-first-bug This seems like it's fixable by a non-expert 6.topic: python 8.has: clean-up and removed 0.kind: bug Something is broken labels Oct 28, 2020
braunse added a commit to braunse/nixpkgs that referenced this issue Oct 28, 2020
Use python3 Sphinx package instead of python2 Sphinx package,
to avoid build-time dependency on python 2, as per
NixOS#101964
@SuperSandro2000
Copy link
Member

 python3.7-aria2p
 python3.7-pygccxml
 python3.7-pyside
 python3.7-pyside-shiboken
 python3.7-pyside-tools
 python3.8-aria2p
 python3.8-pygccxml
 python3.8-pyside
 python3.8-pyside-shiboken
 python3.8-pyside-tools

Why are those in this list? Do they have a python2 variant?

@jonringer
Copy link
Contributor Author

jonringer commented Oct 28, 2020

not entirely sure. It could be that some library they use, uses python2 to do something like generate docs. So they aren't directly using python2 packages.

@primeos
Copy link
Member

primeos commented Oct 29, 2020

Why are those in this list? Do they have a python2 variant?

I assume those are mostly cases of Python 3 packages that depend on some non-Python package that in turn depends on some Python 2 package (e.g. python3Packages.aria2p -> aria2 -> ... -> python27Packages.cryptography).

The following can help to navigate the dependency trees more efficiently (/ to search for the Python 2 cryptography and then h to navigate up in the dependency tree):

$ nix-tree $(nix-instantiate -A python3Packages.aria2p)

Should I try to write a script (or does someone already know/have one) to ping the maintainers of the affected packages?

@SuperSandro2000
Copy link
Member

then h to navigate up in the dependency tree):

The root is on the left and then use vim style key bindings or arrow keys.

@domenkozar
Copy link
Member

cachix fixed in 59c53bc

jonringer pushed a commit that referenced this issue Oct 29, 2020
Use python3 Sphinx package instead of python2 Sphinx package,
to avoid build-time dependency on python 2, as per
#101964
mkenigs added a commit to mkenigs/nixpkgs that referenced this issue Oct 29, 2020
@mkenigs mkenigs mentioned this issue Oct 29, 2020
10 tasks
@jonringer
Copy link
Contributor Author

I added some directions on how to locate cryptography in a dependency tree

jonringer pushed a commit that referenced this issue Oct 30, 2020
@jonringer
Copy link
Contributor Author

oh, nix-tree is way easier to navigate

@jonringer
Copy link
Contributor Author

jonringer commented Oct 30, 2020

I tried packaging asciidoc-py3, but everything they do has xml imports to web urls.... :(

primeos pushed a commit to braunse/nixpkgs that referenced this issue Oct 30, 2020
Asciidoc is severely outdated and is replaced by asciidoctor.
Eliminating asciidoc gets rid of a python2 dependency, see
NixOS#101964
dasJ added a commit to helsinki-systems/nixpkgs that referenced this issue Jan 18, 2021
Main motivation was to port it to Python3 which is due to NixOS#101964.
So:

- Switch from asciidoc to asciidoc-py3
- Switch from Python 2 to Python 3
- This needs autoreconfHook now
- We also need to do some patching in a2x.py
- Switch to patchShebangs (more readable)
- Only input w3m if we actually build with it
zimbatm pushed a commit that referenced this issue Jan 21, 2021
Main motivation was to port it to Python3 which is due to #101964.
So:

- Switch from asciidoc to asciidoc-py3
- Switch from Python 2 to Python 3
- This needs autoreconfHook now
- We also need to do some patching in a2x.py
- Switch to patchShebangs (more readable)
- Only input w3m if we actually build with it
@dotlambda dotlambda added 9.needs: clean-up and removed 3.skill: good-first-bug This seems like it's fixable by a non-expert 3.skill: sprintable A larger issue which is split into distinct actionable tasks 8.has: clean-up labels Mar 8, 2021
@FRidh
Copy link
Member

FRidh commented Mar 25, 2021

Next step would be to make python an alias to python2 in aliases.nix and evaluate Nixpkgs with allowAliases = false;.

@FRidh
Copy link
Member

FRidh commented Mar 25, 2021

Working on this in #117570.

@stale
Copy link

stale bot commented Oct 12, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 12, 2021
@thiagokokada
Copy link
Contributor

thiagokokada commented Nov 28, 2021

Some updates:

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 28, 2021
@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 19, 2022
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 23, 2022
@rnhmjoj
Copy link
Contributor

rnhmjoj commented Nov 23, 2022

The only application left is NixOps, but there already is nixopsUnstable which does use Python 3.
I'd say the issue can now be closed. The next step is #201859.

@rnhmjoj rnhmjoj closed this as completed Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.