End to end approach for third-party packages handlings in ScanCode

[pombredanne]

When this all done, we will have NO files under thirdparty/ anymore.

Instead I will have:

1. all the wheels, source archives, ABOUT files and licenses, etc available for reference as release assets in some public GitHub PyPI-like repository that will be at https://github.com/nexB/third-party-packages/release

2. I have several requirements files with hashes, one per OS/Arch/Python combo for production.

3. I have one requirements file with hashes (supporting for all OS/Arch/Pythons combos) that is used for development only (e.g. it would have the tools we use under thirdparty/dev as well as new libraries such as pip-tools and similar)

4. For each release of ScanCode, I have the following assets attached to the scancode-toolkit release at https://github.com/nexB/scancode-toolkit/releases .This means:

• we have 4 OS/arch combos: linux-x86_64, macOS-x86_64, win-x86_64, win-x86_32
• we 3 CPython versions: 3.6, 3.7, 3.8 (and soon 3.9)
• Therefore we have 4 x 3 = 12 different installable sdist archives that area available in https://github.com/nexB/scancode-toolkit/release

In terms of what I can do:

• Use case: Install self-contained ScanCode archive Use case: Install self-contained ScanCode archive #2090 : Here I download an archive that is specific for
  • one version of scancode
  • one version of Python
  • one combo of os/arch

This archive is an sdist-like archive and I can extract it and run as we can do it today with existing release archive. This archive is one of the 12 possible archives as listed in (4)

• Use case: Install ScanCode from a git checkout Use case: Install ScanCode from a git checkout #2089 : We will ensure that the configure.py script work such such that it no longer uses wheels from thirdparty/ but instead fetches installs wheel from our PyPI repo in (1) using requirement files from (2) and the dev requirement file from (3)

The other tickets are to actually create utilities to support these use case and to support scancode developers to automate the release operations.

• Dev utility: build and download the sdist and wheels of a single package and all its dependencies Dev utility: build and download the sdist and wheels of a single package and all its dependencies #2087

  • and this which is mostly a subticket of Dev utility: build and download the sdist and wheels of a single package and all its dependencies #2087 : Dev utility: Create build loop in repo to build and upload wheels and sdist for all operating systems and Python versions in our "personal" PyPI Dev utility: Create build loop in repo to build and upload wheels and sdist for all operating systems and Python versions in our "personal" PyPI #2071

• Dev utility: Upload to GitHub release Dev utility: Upload to GitHub release #2136 ... this is also used by Dev utility: build and download the sdist and wheels of a single package and all its dependencies #2087

• Dev utility: Create script to generate requirements with hashes Dev utility: Create script to generate requirements with hashes skeleton#50 in PR Automate the creation of requirement.txt and fetch deps #2118

• Dev utility: Create release scripts to build archives Dev utility: Create release scripts to build archives #2072 part of PR Expose all .whl,.ABOUT, .NOTICE,.LICENCE files and Create an archive #2117

• Dev utility: Improve configure script to work of pinned requirements Dev utility: Improve configure script to work of pinned requirements #2070

• Dev utility: Create new repo to expose pre-built wheels, tarballs and ABOUT files as releases in a PyPI-like repo Dev utility: Create new repo to expose pre-built wheels, tarballs and ABOUT files as releases in a PyPI-like repo #2068

[Abhishek-Dev09-zz]

@pombredanne :
Direction for using new provision of thirdparty:

• Activate environment for python 3 and run pip install -r etc/scripts/req_tools.txt in terminal.

• Upload all pre-built wheels, tarballs and ABOUT files as releases in a PyPI-like repo like my repo . This can be used in --find-links option.
  For example - github_release.py --user USER --repo REPO --tag-name TAG_NAME
  --directory DIRECTORY [--token TOKEN]
  [--description DESCRIPTION]
  [--retry_limit RETRY_LIMIT]

• Produce requirement files

  1. To produce for all Oses for python 3.6
     Run python etc/scripts/freeze_and_update_reqs.py --find-links github_pypi --requirement requirements.txt

  2. To produce for OS/python
     a)Run python3 etc/scripts/deps_archive.py --find-links github_pypi --requirement requirements.txt --archive-name macOS_py36
     b)Run python etc/scripts/freeze_and_update_reqs.py --find-links macOS_py36 --requirement macOS_req_py36.txt

• Link configure.py with github_pypi/macOS_py36(deps_only_achive) via requirements file.
  This must contains pip-sync macOS_req_py36 --find-links github-pypi/macOS_py36 . This will install/upgrade/uninstall
  everything necessary to match the requirements.txt contents. (Todo)

• Replace thirdparty with macOS_py36 in manifest.in and MANIFEST.in.release . Then it is ready for release.Also work defining proper name to scancode archive such as scancode_macOS_py36 , scancode_win32_py36 . (To Do)

• Use romp and pip dowload for build wheel for all oses/python . (To do)

[pombredanne]

That looks good! Can you also try that on Linux and Python 3.7 too at least?

[Abhishek-Dev09-zz]

@pombredanne if you place python 37/38 wheels from here to thirdparty . It will work on linux .

[Abhishek-Dev09-zz]

When we run pip-compile ,the generated Requirement.txt is missing some package which is defined by python_version in setup.py such as ftty.
For example:

'ftfy <  5.0.0; python_version == "2.7"',
'ftfy>=  5.0.0; python_version > "3"',

See this issue at jazzband/pip-tools#1202

[pombredanne]

Eventually we built a tool that handles creation, documentation and publishing of wheels and source https://github.com/nexB/scancode-toolkit/tree/1bddb92c70abcf26cf9435061e429bae62cbe040/etc/release
And this is also part of the https://github.com/nexB/skeleton

Using romp it builds wheel and tests them on many OS/Python combos

In the end they end up pushed to https://thirdparty.aboutcode.org/pypi/