We recognize that the pipenv release process is currently poorly managed. That's why this document seeks to streamline the release process, identify current complexities, and help eliminate existing single points of failure.
Any new release will require you to have the latest version of master installed in development mode:
$ git checkout master
$ git fetch origin
$ git pull
$ pipenv install --devA lot of pipenv depends on ancillary libraries. You may need to make new releases of:
This is the most complex element of releasing new versions of pipenv due to existing patches on current code. Currently the largest patchsets are maintained against pip and pip-tools.
You can begin by reviewing vendored dependencies which can be found in pipenv/vendor/vendor.txt, a file which is consumed by the automated vendoring process. These dependencies may have minor patches applied which can be found in tasks/vendoring/patches/vendor. Check PyPI for updates to the specified packages and increment the versions as needed, making sure to capture all dependencies in case any were added. It would be very bad to release without necessary dependencies, obviously.
Next you can consult pipenv/patched/patched.txt which enumerates the patched dependencies. Follow the same process, but be aware that you will need to rewrite patches for each dependency once you update (most likely) as they do tend to change somewhat substantially.
Pipenv also includes a vendored copy of safety for checking for vulnerabilities against the pyup.io database. In order to update the safety package, run the following:
$ inv vendoring.update-safetyFor larger libraries you can keep local clones of them and simply generate full patch sets in which you replace the updated path in pipenv when you are done making changes. Here is an example of a script used from inside a local clone of pip to generate a patch and copy it to pipenv's local patches directory.
#!/usr/bin/bash
sed -i -r 's/([a-b]\/)(?:src\/)?(pip)/\1pipenv\/patched\/\2/g' diff.patch
cp diff.patch ../pipenv/tasks/vendoring/patches/patched/pip19.patchAssuming patches are kept up to date and you are simply working on a modification that is relatively minor, here is a script you can use to pull in a single modified file from pipenv into a local clone of pip-tools and regenerate a patch from the updated version of that file:
#!/bin/bash
cp ../pipenv/pipenv/patched/piptools/$@ piptools/$@
sed -i 's/pipenv.patched.notpip/pip/g' piptools/$@
git diff -p piptools/$@ > diff.patch
sed -i -r 's:([a-b]/)(piptools):\1pipenv/patched/\2:g' diff.patchThe resulting patch is then combined into the patchset by hand (make sure you don't alter the whitespace in the patch!)
Okay, now that's done, it's time to update vendored dependencies. You can install pipenv itself by moving to the source directory (cd pipenv) and running pip install -e .. Then you can run pipenv install --dev to install the development dependencies into a virtual environment.
Update the vendored dependencies by copying the pipenv/vendor/vendor.txt file to a new directory (e.g. /tmp/vendor) and unpinning all of the dependencies. Note there is a helper script for this:
$ pipenv run inv vendoring.unpin-and-update-vendoredThis should unpin all vendored and patched dependencies and resolve them; ideally you would keep the file formatted so that we can see what depends on what, but this will tell you what can be updated & provide the latest versions.
To re-vendor and patch the vendored libraries, run the command:
$ pipenv run inv vendoring.updateThis will automatically remove the ./pipenv/vendor/ and ./pipenv/patched/ directories and re-download and patch the specified dependencies. It will also attempt to download any relevant licenses. Once this is completed, run git status and inspect the output -- look through the git diff for anything that may cause breakages. If any licenses have been deleted, you will need to determine why they were not replaced by the license download tooling.
Make sure to read through any modified license files for changes -- note that we cannot redistribute code that is licensed under a copyleft license, such as the GPL. Similarly, all vendored code must be licensed or it cannot be redistributed. If vendored libraries have become unlicensed or are no longer usable, suitable replacements will have to be found and potentially patched into the vendored dependencies. This may be a good time to consider simply including the dependency as an install requirement.
Look into using a tool like https://fossa.com/ to help with this.
Now we will need to update the lockfile. This is required to ensure tests run against the latest versions of libraries. You will need to run the following:
# use the latest python here
$ export PIPENV_PYTHON=3.8
$ pipenv lock --dev
# Inspect the changes in a diff viewer, for example we should keep the python 2 dependencies to use for running tests
# on completion, stage the relevant changes
$ export PIPENV_PYTHON=2.7
$ pipenv lock --keep-outdated --dev
# this helps avoid overwriting the entire lockfile and should introduce only the changes required to run tests on python 2
# inspect the resulting lockfile and commit the changes
$ git commitTest pipenv locally. If tests pass, you can go ahead and make a PR to merge whatever you want to release.
$ export PIPENV_PYTHON=3.8
$ pipenv install --dev && pytest -ra tests
$ export PIPENV_PYTHON=2.7
$ pipenv install --dev && pytest -ra testsPipenv now leverages sphinxcontrib.spelling to help ensure documentation does not contain typographical mistakes. To validate documentation, please make sure to rebuild and rectify any documentation issues before pushing the new release:
$ pipenv shell
$ cd docs
$ make clean && make html
$ make spellingValidate the results, adding any new exceptions to docs/spelling_wordlist.txt.
- Set a version:
pipenv run inv release.bump-version --trunc-month --pre --tag=a- this will truncate the current month, creating an alpha pre-release, e.g.2020.4.1a1a. Note: You can pass--tag=bor--tag=rchere as well make check- This has the side-effect of producing wheelsmake tests- Runs tests locallymake upload-test- This uploads artifacts to test-pypi- Consume the version on test pypi, ensure that version is functioning.
- Push the pre-release to github & wait for CI to pass
- Create a new tag, e.g.
v2020.4.1a1and push it to github -- this can be achieved viapipenv run inv release.tag-version --push - The github action will automatically build and push the prerelease to
PyPI - Once a release is pushed, the action will update
masterwith a newdevversion - Review any pull requests and issues that should be resolved before releasing the final version
- The process is identical for releasing a standard release, except the
release.bump-versioncommand is called without any arguments.
If in doubt, follow the basic instructions below.
- Get set up on Test PyPI
- Use Test PyPI to upload the package, make sure the
READMErenders, test that it installs okay, and so on - Get credentials to co-maintain the pipenv project on PyPI.org -- SPOF alert
- Set the version number to a pre-release identifier
- Package and upload pipenv to PyPI as a pre-release/alpha
- Publicize on distutils-sig, Discourse, and the relevant GitHub issue(s) a. write up diplomatic notification
- Recruit manual testing (example) for workflows we don't account for
- Wait a week, then update version number to a canonical release and re-release on PyPI.org
- Publicize on lists, Discourse, GitHub issues
Most of the pipenv related ecosystem libraries are using GitHub actions to automate releases when tags are pushed. Most likely we will look to move in this direction and simplify the process.