Rework packaging scripts to remove pipenv #43

redshiftzero · 2019-05-18T04:47:19Z

Closes #42. I've tried to be pretty verbose here in comments in the diff for maintenance sake.

One can test these changes by going through the full packaging workflow with the following two pending PRs:

remove Pipenv in favor of pip-tools, security updates securedrop-proxy#33 (with commit f946fe38f4dc0b4b71cff3ea8d318fcca1599c09 dropped which fixes the CVEs in the dependencies in that repo (alternatively one can add those packages to FPF's PyPI after review))
remove Pipenv in favor of pip-tools, security updates securedrop-client#372 (with commit 97cdd30117aa0e4dff331d586376a8da6e0eb9f4 dropped which fixes the CVEs in the dependencies in that repo (alternatively one can add those packages to FPF's PyPI after review))

Note: this PR is ready for review and will be taken off of draft mode once this and the above two PRs are approved - this is such that we can merge all three in a coordinated manner without breaking debian package builds

(CI is "failing" because I enabled circle CI here to add CI jobs in #44 but there's no config file in the master branch/this PR, so we should ignore until we merge #44)

packaged repos will have the requirements file called build-requirements.txt

kushaldas

make requirements is changing the requirement files without any hashes.

$ git diff
diff --git a/build-requirements.txt b/build-requirements.txt
index 2cbb1ca..618ebb6 100644
--- a/build-requirements.txt
+++ b/build-requirements.txt
@@ -1,15 +1,15 @@
-alembic==1.0.2 --hash=sha256:14024bd47f71d8b51920721dcd63248d07d370fbd0f6afa9bec67b9edaf71f36
-arrow==0.12.1 --hash=sha256:5ef4a593615dc61ed85e62070b1bd27c71f7266233f0f9f385b651370e8c6760
-certifi==2018.10.15 --hash=sha256:a5471c55b011bd45d6155f5c3629310c1d2f1e1a5a899b7e438a223343de583d
-chardet==3.0.4 --hash=sha256:9f178988ca4c86e8a319b51aac1185b6fe5192328eb5a163c286f4bf50b7b3d8
-idna==2.7 --hash=sha256:954e65e127d0433a352981f43f291a438423d5b385ebf643c70fd740e0634111
-mako==1.0.7 --hash=sha256:87ee3f74ba3ea544e683a5a22e7e34f4d1cf3ad34414b5f3858becf00facf1d6
-markupsafe==1.0 --hash=sha256:6a7078a2fb4406458d6ae3579e4eb01a9bdc0a9a0686a28fa50c19a039e3fcb8
-pathlib2==2.3.2 --hash=sha256:8e276e2bf50a9a06c36e20f03b050e59b63dfe0678e37164333deb87af03b6ad
-python-dateutil==2.7.5 --hash=sha256:56f285e7fad54cde3e31dc68a31a861543bfee5ada9278da8e85ec20a8f72912
-python-editor==1.0.3 --hash=sha256:44fc57a6db6e04c7922c37a04d0a86d0024a4f0f06245b6c57638cb322176202
-requests==2.20.0 --hash=sha256:2a539dd6af40a611f3b8eb3f99d3567781352ece1698b2fab42bf4c2218705b5
-securedrop-sdk==0.0.8 --hash=sha256:116e4461efcd3d0e6b18161a6908cec614f53fb04a5b7ef68f19207559b34d91
-six==1.11.0 --hash=sha256:4663c7a1dbed033cfb294f2d534bd6151c0698dc12ecabb4eaa3cb041d758528
-sqlalchemy==1.2.13 --hash=sha256:7f4fb23a839ff587fb5661caa7fb3f4151d43c641930a1af7559b7b21d91d9db
-urllib3==1.24 --hash=sha256:f31f3342445f7d635a8d473b4741111fd5e7602a7b2b8b60237b7328878b4b88
+alembic==1.0.2 \
+arrow==0.12.1 \
+certifi==2018.10.15 \
+chardet==3.0.4 \
+idna==2.7 \
+mako==1.0.7 \
+markupsafe==1.0 \
+pathlib2==2.3.2 \
+python-dateutil==2.7.5 \
+python-editor==1.0.3 \
+requests==2.20.0 \
+securedrop-sdk==0.0.9 \
+six==1.11.0 \
+sqlalchemy==1.3.3 \
+urllib3==1.24.3 \
diff --git a/requirements.txt b/requirements.txt
index 1005856..618ebb6 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,44 +1,15 @@
-#
-# This file is autogenerated by pip-compile
-# To update, run:
-#
-#    pip-compile --generate-hashes --output-file requirements.txt requirements.in
-#
 alembic==1.0.2 \
-    --hash=sha256:04bcb970ca8659c3607ddd8ffd86cc9d6a99661c9bc590955e8813c66bfa582b
 arrow==0.12.1 \
-    --hash=sha256:a558d3b7b6ce7ffc74206a86c147052de23d3d4ef0e17c210dd478c53575c4cd
 certifi==2018.10.15 \
-    --hash=sha256:339dc09518b07e2fa7eda5450740925974815557727d6bd35d319c1524a04a4c \
-    --hash=sha256:6d58c986d22b038c8c0df30d639f23a3e6d172a05c3583e766f4c0b785c0986a
 chardet==3.0.4 \
-    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
-    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691
 idna==2.7 \
-    --hash=sha256:156a6814fb5ac1fc6850fb002e0852d56c0c8d2531923a51032d1b70760e186e \
-    --hash=sha256:684a38a6f903c1d71d6d5fac066b58d7768af4de2b832e426ec79c30daa94a16
 mako==1.0.7 \
-    --hash=sha256:4e02fde57bd4abb5ec400181e4c314f56ac3e49ba4fb8b0d50bba18cb27d25ae
 markupsafe==1.0 \
-    --hash=sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665
 pathlib2==2.3.2 \
-    --hash=sha256:8eb170f8d0d61825e09a95b38be068299ddeda82f35e96c3301a8a5e7604cb83 \
-    --hash=sha256:d1aa2a11ba7b8f7b21ab852b1fb5afb277e1bb99d5dfc663380b5015c0d80c5a
 python-dateutil==2.7.5 \
-    --hash=sha256:063df5763652e21de43de7d9e00ccf239f953a832941e37be541614732cdfc93 \
-    --hash=sha256:88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02
 python-editor==1.0.3 \
-    --hash=sha256:a3c066acee22a1c94f63938341d4fb374e3fdd69366ed6603d7b24bed1efc565
 requests==2.20.0 \
-    --hash=sha256:99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c \
-    --hash=sha256:a84b8c9ab6239b578f22d1c21d51b696dcfe004032bb80ea832398d6909d7279
 securedrop-sdk==0.0.9 \
-    --hash=sha256:43146f02c41858578f7c9997e733f2a07f8a4877f1bf9f8b4b11fd4ceffa47a9
 six==1.11.0 \
-    --hash=sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9 \
-    --hash=sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb
 sqlalchemy==1.3.3 \
-    --hash=sha256:91c54ca8345008fceaec987e10924bf07dcab36c442925357e5a467b36a38319
 urllib3==1.24.3 \
-    --hash=sha256:2393a695cd12afedd0dcb26fe5d50d0cf248e5a66f75dbd89a3d4eb333a61af4 \
-    --hash=sha256:a637e5fae88995b256e3409dc4d52c2e2e0ba32c42a6365fee8bbd2238de3cfb

make build-wheels is thus failing. Example:

$ PKG_DIR=/home/user/code/securedrop-client make build-wheels
./scripts/fetch-wheels
WARNING: Using dev wheels location for build...
Fetching wheel URLs... 44/44... done.
./scripts/verify-sha256sum-signature
gpg: key F08893B959CAB065: 6 signatures not checked due to missing keys
gpg: key F08893B959CAB065: public key "Conor Schaefer <[email protected]>" imported
gpg: key DA05B7C52ABAF334: 8 signatures not checked due to missing keys
gpg: key DA05B7C52ABAF334: public key "Jennifer Helsby <[email protected]>" imported
gpg: key D8219C8C43F6C5E1: 55 signatures not checked due to missing keys
gpg: key D8219C8C43F6C5E1: public key "Kushal Das <[email protected]>" imported
gpg: key AAFB3575FAC82745: 1 signature not checked due to a missing key
gpg: key AAFB3575FAC82745: public key "Mickael E. <[email protected]>" imported
gpg: key 699C99DC7131FD81: 6 signatures not checked due to missing keys
gpg: key 699C99DC7131FD81: public key "Michael Sheinberg <[email protected]>" imported
gpg: Total number processed: 5
gpg:               imported: 5
gpg: no ultimately trusted keys found
gpg: Signature made Mon 06 May 2019 09:35:06 PM IST
gpg:                using RSA key 0BC135125EB2FF9A0F88EE1CC65FF007C75766ED
gpg: Good signature from "Kushal Das <[email protected]>" [unknown]
gpg:                 aka "Kushal Das <[email protected]>" [unknown]
gpg:                 aka "Kushal Das <[email protected]>" [unknown]
gpg:                 aka "Kushal Das <[email protected]>" [unknown]
gpg:                 aka "Kushal Das <[email protected]>" [unknown]
gpg:                 aka "Kushal Das <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A85F F376 759C 994A 8A11  68D8 D821 9C8C 43F6 C5E1
     Subkey fingerprint: 0BC1 3512 5EB2 FF9A 0F88  EE1C C65F F007 C757 66ED
Checking that SHA256SUMs from S3 match signed file...  OK
./scripts/build-sync-wheels -p /home/user/code/securedrop-client
Invalid requirement: 'alembic==1.0.2 arrow==0.12.1 certifi==2018.10.15 chardet==3.0.4 idna==2.7 mako==1.0.7 markupsafe==1.0 pathlib2==2.3.2 python-dateutil==2.7.5 python-editor==1.0.3 requests==2.20.0 securedrop-sdk==0.0.9 six==1.11.0 sqlalchemy==1.3.3 urllib3==1.24.3'
Traceback (most recent call last):
  File "/usr/share/python-wheels/packaging-16.8-py2.py3-none-any.whl/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 1617, in parseString
    raise exc
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 1607, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 3376, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 1383, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pyparsing-2.1.10-py2.py3-none-any.whl/pyparsing.py", line 3164, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected stringEnd (at char 15), (line:1, col:16)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 82, in __init__
    req = Requirement(req)
  File "/usr/share/python-wheels/packaging-16.8-py2.py3-none-any.whl/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'arrow==0'"

Traceback (most recent call last):
  File "./scripts/build-sync-wheels", line 84, in <module>
    main()
  File "./scripts/build-sync-wheels", line 50, in main
    subprocess.check_call(cmd)
  File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip3', 'download', '--no-binary', ':all:', '--require-hashes', '-d', '/tmp/tmpambqkc_5', '-r', '/home/user/code/securedrop-client/requirements.txt']' returned non-zero exit status 1
Makefile:36: recipe for target 'build-wheels' failed
make: *** [build-wheels] Error 1

kushaldas · 2019-05-20T06:21:05Z

I tried the above against bye-pipenv branch of securedrop-client repo.

redshiftzero · 2019-05-20T20:19:37Z

Thanks for running through this. That requirements.txt state could happen previously if make requirements failed for some reason, e.g. a dependency was absent from FPF's PyPI mirror. I've added two commits which:

Won't modify the input requirements file unless update-requirements succeeds
Removes create-requirements and just does all the relevant logic in one place in Python in update-requirements to simplify a bit

Let me know what you think! This should do the trick.

kushaldas

The following dependent wheel(s) are missing:
python-dateutil==2.7.5
python-editor==1.0.3

It is still marking currently available wheels as missing.

kushaldas · 2019-05-27T07:31:30Z

I will push the change of the requirements.txt file name to this PR.

redshiftzero · 2019-05-29T00:28:29Z

I've taken all three PRs out of draft mode for merge:

#43 (this PR)
freedomofpress/securedrop-client#372
freedomofpress/securedrop-proxy#33

kushaldas

This is now approved.

redshiftzero added 4 commits May 17, 2019 21:37

build-requirements.txt <-> requirements.txt

2087557

packaged repos will have the requirements file called build-requirements.txt

verify-hashes: update assuming format is requirements.txt w/hashes

2019809

update-requirements: modify script for requirements.txt format

89c0832

build-sync-wheels: simplify since we use the requirements file directly

3f79e6a

redshiftzero requested a review from kushaldas May 18, 2019 04:47

fix incorrect path to alembic migration

ffda61d

kushaldas suggested changes May 20, 2019

View reviewed changes

redshiftzero mentioned this pull request May 20, 2019

Add pipeline for building Debian packages in CircleCI #44

Merged

redshiftzero added 2 commits May 20, 2019 12:39

update-requirements: don't write out intermediate requirements files

254afba

simplify by removing create-requirements

fb18301

kushaldas suggested changes May 21, 2019

View reviewed changes

Fix wheels incorrectly marked as absent due to _ versus -

a2142c6

Adds new requirements file name for debian packaging

ca9fab4

redshiftzero marked this pull request as ready for review May 29, 2019 00:27

kushaldas approved these changes May 30, 2019

View reviewed changes

kushaldas merged commit 8f77fdb into master May 30, 2019

redshiftzero mentioned this pull request Jun 8, 2019

Update release manager steps to account for pipenv removal #52

Merged

sssoleileraaa deleted the bye-pipenv branch September 13, 2021 23:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework packaging scripts to remove pipenv #43

Rework packaging scripts to remove pipenv #43

redshiftzero commented May 18, 2019 •

edited

Loading

kushaldas left a comment

kushaldas commented May 20, 2019

redshiftzero commented May 20, 2019

kushaldas left a comment

kushaldas commented May 27, 2019

redshiftzero commented May 29, 2019

kushaldas left a comment

Rework packaging scripts to remove pipenv #43

Rework packaging scripts to remove pipenv #43

Conversation

redshiftzero commented May 18, 2019 • edited Loading

kushaldas left a comment

Choose a reason for hiding this comment

kushaldas commented May 20, 2019

redshiftzero commented May 20, 2019

kushaldas left a comment

Choose a reason for hiding this comment

kushaldas commented May 27, 2019

redshiftzero commented May 29, 2019

kushaldas left a comment

Choose a reason for hiding this comment

redshiftzero commented May 18, 2019 •

edited

Loading