Crash when making virtualenv directory with parallel pipenv invocations

[swgillespie]

Issue description

When running multiple pipenv processes simultaneously, they may both attempt to use 'mkdir_p' at the same time, which can cause one of the pipenv processes to crash when it tries to create a directory that already exists. This is a classic TOCTOU bug that has bitten many a piece of software.

I personally hit this in my continuous integration setup where I invoke pipenv multiple times, potentially in parallel, and see crashes such as this:

[ tes/examples/python/smoke-test ] sample: /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/examples/python/smoke-test
[ etes/examples/python/guestbook ] sample: /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/examples/python/guestbook
[ tes/examples/python/smoke-test ] pulumi: /home/travis/.pulumi/bin/pulumi
[ etes/examples/python/guestbook ] pulumi: /home/travis/.pulumi/bin/pulumi
[ tes/examples/python/smoke-test ] **** Invoke '/home/travis/.local/bin/pipenv --python 3' in '/tmp/p-it-travis-job-smoke-test-aced7882-102623154'
[ etes/examples/python/guestbook ] **** Invoke '/home/travis/.local/bin/pipenv --python 3' in '/tmp/p-it-travis-job-guestbook-63f89532-423054697'
[ etes/examples/python/guestbook ] Invoke '/home/travis/.local/bin/pipenv --python 3' failed: exit status 1
[ etes/examples/python/guestbook ] Traceback (most recent call last):
[ etes/examples/python/guestbook ]   File "/home/travis/.local/bin/pipenv", line 11, in <module>
[ etes/examples/python/guestbook ]     sys.exit(cli())
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
[ etes/examples/python/guestbook ]     return self.main(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
[ etes/examples/python/guestbook ]     rv = self.invoke(ctx)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
[ etes/examples/python/guestbook ]     return Command.invoke(self, ctx)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
[ etes/examples/python/guestbook ]     return ctx.invoke(self.callback, **ctx.params)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
[ etes/examples/python/guestbook ]     return callback(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
[ etes/examples/python/guestbook ]     return ctx.invoke(f, obj, *args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
[ etes/examples/python/guestbook ]     return callback(*args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
[ etes/examples/python/guestbook ]     return f(get_current_context(), *args, **kwargs)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/cli/command.py", line 203, in cli
[ etes/examples/python/guestbook ]     clear=state.clear,
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/core.py", line 565, in ensure_project
[ etes/examples/python/guestbook ]     pypi_mirror=pypi_mirror,
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/core.py", line 483, in ensure_virtualenv
[ etes/examples/python/guestbook ]     if not project.virtualenv_exists:
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 259, in virtualenv_exists
[ etes/examples/python/guestbook ]     if self.pipfile_exists and os.path.exists(self.virtualenv_location):
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 398, in virtualenv_location
[ etes/examples/python/guestbook ]     self._virtualenv_location = self.get_location_for_virtualenv()
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 272, in get_location_for_virtualenv
[ etes/examples/python/guestbook ]     name = self.virtualenv_name
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 384, in virtualenv_name
[ etes/examples/python/guestbook ]     sanitized, encoded_hash = self._get_virtualenv_hash(self.name)
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/project.py", line 363, in _get_virtualenv_hash
[ etes/examples/python/guestbook ]     or get_workon_home().joinpath(venv_name).exists()
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/utils.py", line 1254, in get_workon_home
[ etes/examples/python/guestbook ]     mkdir_p(str(expanded_path))
[ etes/examples/python/guestbook ]   File "/home/travis/.local/lib/python2.7/site-packages/pipenv/utils.py", line 571, in mkdir_p
[ etes/examples/python/guestbook ]     os.mkdir(newdir)
[ etes/examples/python/guestbook ] OSError: [Errno 17] File exists: '/home/travis/.local/share/virtualenvs'

The root cause is that two pipenv processes are racing to create the new directory subtree rooted at /home/travis/.local/share/virtualenvs, one of them wins, and the other crashes.

Expected result

I'd except both pipenv processes to be successful.

Actual result

The pipenv that loses the race to create the directory crashes.

Steps to replicate

Launch multiple instances of pipenv --python 3 on a machine that previously has not used pipenv to create any virtualenvs. If you do this enough times, you'll hit the race and crash.

[duplicate-issues]

Hey @swgillespie,

We did a quick check and this issue looks very darn similar to

• #3256 - Fix parsing of named `setup.py` extras
• #3255 - Installation failed due to missing hashes when using --skip-lock
• #3252 - Updated graph command docstrings, fixes #3071

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

[swgillespie]

Fixed by #3258.