Use symlinks to add dists in the Pex CLI.

[jsirois]

This avoids copying dists from ~/.pex/installed_wheels to a PEXBuilder
tmpdir before finally zipping up the tmpdir. Instead, with just a single
symlink added to the PEXBuilder tmpdir for each dist, the zip is created
directly from the sources under ~/.pex/installed_wheels providing a
modest speed up when building large PEX files.

[jsirois]

For tc.pex which has 114 distributions and weighs in at 48M - making it a moderate sized PEX - I found:

$ hyperfine --prepare 'true' --prepare 'git stash pop || true' 'python -mpex -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository.pex --pex-repository tc.pex' 'python -mpex -v -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository-symlinked.pex --pex-repository tc.pex'
Benchmark #1: python -mpex -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository.pex --pex-repository tc.pex
  Time (mean ± σ):      9.622 s ±  0.093 s    [User: 8.466 s, System: 1.066 s]
  Range (min … max):    9.525 s …  9.774 s    10 runs
 
Benchmark #2: python -mpex -v -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository-symlinked.pex --pex-repository tc.pex
  Time (mean ± σ):      8.139 s ±  0.084 s    [User: 7.635 s, System: 0.469 s]
  Range (min … max):    8.047 s …  8.290 s    10 runs
 
Summary
  'python -mpex -v -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository-symlinked.pex --pex-repository tc.pex' ran
    1.18 ± 0.02 times faster than 'python -mpex -r ~/dev/toolchainlabs/toolchain/3rdparty/python/requirements.txt --constraints ~/dev/toolchainlabs/toolchain/3rdparty/python/constraints.txt -o tc-via-pex-repository.pex --pex-repository tc.pex'

[jsirois]

The new SYMLINK copy_mode brought to light the fact we were doing over-writes here when populating the PEXBuilder with the results of multi-target resolves whenever a distribution applied to more than one target (say a universal wheel). This was a harmless time-waste previously.

[Eric-Arellano]

Thanks!