Upgrade to support also Python 3.x

[pombredanne]

The time is now!
Some important reads:

• the official guide: https://docs.python.org/3/howto/pyporting.html
• a great extensive guide from @fedora-python https://portingguide.readthedocs.io/en/latest/process.html

[pombredanne]

And this likely means only Python 3.5 and up

[sschuberth]

Here's one more vote to support Python 3 as that would make debugging multiprocessing exceptions like issue #413 a bit easier.

[pombredanne]

At a high level the key issues I can think of are:

• unicode vs. bytes, since a good many functions handle low level bytes and not unicode and Py3 is unicode by default.
• native libraries using C/C++
• other third-party libraries
• doctests (with u'' in their expectations) or other similar tests

And in all cases, I want to support 2 and 3 at the same time with a single codebase which is not a big deal.

FWIW, most libraries should be OK except some natives. Upcoming ones (e.g. https://github.com/nexB/license-expression/ ) are also ok (I just ported that to Python 3)

[pombredanne]

For the 2 to 3 update, the first step will be effectively to gather wheels for every ScanCode-bundled dependencies

I do not know which one would be OK or not: this has to be tested

And eventually a Python 3 wheel --if not available-- can be built for all OSes/archs using https://github.com/pombreda/thirdparty-manylinux/ https://github.com/pombreda/thirdparty that are Travis and Appveyor build loops that create wheels and upload them to http://dl.bintray.com/pombreda/thirdparty/

Note that I have already started collecting Python 36 wheels there

And once all deps are available, the second step is to ensure that all the tests pass both on Py2 and Py3 and start making fixes, one at a time which will be the guide to make the port work

And BTW the minimum Python3 version we will support in ScanCode is 3.6.1.
Nothing before that 😉

[vivonk]

According to me, most helpful is this so why we are collecting python wheels on version dependent. Are you trying say that packages of python3 should be installed as well like it do in python 2 case(bin/python2.7) for virtualenv purposes?

[pombredanne]

@vivonk yes, that should be the way

[sschuberth]

More reasons to upgrade to Python 3.x, and version 3.7 in particular: https://docs.python.org/3/whatsnew/3.7.html#optimizations

[davidak]

You have 11 months left to migrate to python 3. Then python 2 will no longer be maintained.

https://pythonclock.org/

What is the status?

[pombredanne]

@davidak this is definitely in the plan for this year!
This is likely quite a bit of work especially if we want to keep support for Python 2 which may not be a good idea?

Help is definitely needed and welcomed. Would you be willing to chip in to help a bit if you have a few spare cycles?

Also, we are planning to participate in the Google Summer of Code this year and if we are accepted that would be a great student project. Mentoring help could be great there.

[sschuberth]

This is likely quite a bit of work especially if we want to keep support for Python 2 which may not be a good idea?

Wouldn't it be a nice & clean cut to make ScanCode 3.0 only support Python 3? That would be easy to remember 😉

[davidak]

Would you be willing to chip in

Sadly i have already too many projects :D

[pombredanne]

@davidak fair enough: may be you will be willing to kick in the tires when we start to have something usable?

[davidak]

For now it works for me to manually set the license in my script. When this library works with python 3 i might try to detect it automatically.

[pombredanne]

from @dankegel in #1342

can't install with pip3
python2.7 is dying (see https://pythonclock.org/ ), time to make sure python3 support is happy.

On Ubuntu 18.04, installing with 'pip3 install scancode-toolkit' fails with

Collecting scancode-toolkit
 Could not find a version that satisfies the requirement scancode-toolkit (from versions: )
No matching distribution found for scancode-toolkit

This is especially annoying since 'pip install scancode-toolkit' fails with a sigsegv (not scancode's fault; 'pip install simplejson' also crashes. See simplejson/simplejson#114 for workaround).

[pombredanne]

I wrote before

And in all cases, I want to support 2 and 3 at the same time with a single codebase which is not a big deal.

And

This is likely quite a bit of work especially if we want to keep support for Python 2 which may not be a good idea?

Unless someone disagree, the next major version after the 3.x release (4.x) will be ported and made to run only on Python3. And Python2 support will be dropped in 4.x and available only in maintenance branches of the 3.x releases.

Having a codebase that does not try to support both versions will help quite a bit. Also I do not want to try to support any Python 3 version before version 3.6. So it is more like we will be migrating to Python 3.6+

Objections and feedback welcomed!

[dankegel]

Happy to lose python 2.7. python's story is way too hairy at the moment; we need to get to a place where everyone can always use venv.

[mjherzog]

Python 3.6 sounds good

[steven-esser]

I agree with @dankegel. Once we are fully migrated to python36+ I do not see a good reason to go out of our way to support python2.

[pombredanne]

We now support fully Python 3.6, 3.7, 3.8 and 3.9 on Linux, Windows and macOS 64 bits.
we dropped Python 2 support (available in legacy 3.x versions)
Thank you all for the efforts and @Abhishek-Dev09 in particular!