Requirements generated with pip-tools > 5.0.0 not parseable by build scripts

[emkll]

Initially observed in freedomofpress/securedrop-proxy#82

When generating a requirements file using pip-tools > 5, the syntax is slightly different, and cannot be parsed by our build scripts, and returns the following error:

Missing sha256sum for package: #
make: *** [Makefile:5: securedrop-proxy] Error 1

Reverting to the < 5 pip-tools produces a requirement.txt file that is parseable by the script. The following diff illustrate difference in format of the requirements file (this is the diff from >5 to <5 as an example)

We should :
0. Investivate why/when requirements.txt is parsed in the context of package building? the dh_virtualenv override should mean that we don't need to touch requirements.txt (https://github.com/freedomofpress/securedrop-debian-packaging/blob/bafccb44db24435d587ba75446eaf16e36f4070f/securedrop-proxy/debian/rules#L17)

1. lock pip-tools in the requirements file if possible (or use a distribution-provided version - though I can't find pip-tools for buster
2. Ensure the build logic can handle both formats of requirements files, if possible

[kushaldas]

The following diff illustrate difference in format of the requirements file (this is the diff from >5 to <5 as an example)

Can you please paste one such diff?

For me in Debian Buster with pip-tools==6.0.1, the requirements.txt file is exactly the same with develop of the securedrop-proxy. I am sure I am missing a step.

[emkll]

@kushaldas here is the difference i am observing https://github.com/freedomofpress/securedrop-proxy/compare/c1b8e30..a5da1db

The first commit was generated with the more recent version of pip-tools, the latter with an older version (apologies i forgot to capture the exact versions).

[eloquence]

(On current sprint for tracking purposes to monitor if we still encounter this issue with recent changes to the build process.)

[sssoleileraaa]

note that we'll keep running into this issue when we run make update-pip-requirements if we want to use pip-tools > 5.0.0 for dev or test requirements. we could downgrade pip-tools in the makefile before running pip-compile, just for requirements.txt as a solution.

[sssoleileraaa]

^ see freedomofpress/securedrop-proxy#88 for a workaround for this issue until it is fixed, specifically addressed here: https://github.com/freedomofpress/securedrop-proxy/blob/03615733d44d34e49a3c2eb339fe9802636c609a/Makefile#L58

this allows us to run make update-pip-dependencies with the latest pip-tools for dev requirements

[sssoleileraaa]

I haven't looked into a direct fix yet, but figured we could check in next week before one of us begins investigating

[sssoleileraaa]

@emkll #248 should fix this issue, which will allow me to remove my workaround in freedomofpress/securedrop-proxy#88 and unblock other PRs