Add --python-executable option for pip-sync

[MaratFM]

Problem: build pipelines and infrastructure at big companies often targets multiple independent python projects. Currently, in order to use pip-sync, pip-tools have to be installed in every environment. When pip-compile does not have such restriction.

Solution: add the ability to pass custom python executable. When the option provided, list of installed packages will be collected from the target environment and all install/uninstall commands will be executed inside that environment.

This PR implements a proof concept for collecting early feedback on the idea and implementation. If concept and task will be accepted, sufficient test coverage will be added and PR re-submitted.

Thank you!

Added "--python-executable" option for pip-sync command. It allows targeting environments other than current:

Contributor checklist

• Provided the tests for the changes.
• Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[MaratFM]

@atugushev could you please comment if this PR makes sense and maintainers agree to add such an option

[atugushev]

Hello @MaratFM,

Tganks for pinging! I believe that this functionality would help for such issues as #1087. So i'm 👍 on this.

[atugushev]

Generally, it looks good. One thing to consider: could we improve --python-executable to pass binary name somehow? For example: pip-sync --python-executable=python3.

[MaratFM]

Generally, it looks good. One thing to consider: could we improve --python-executable to pass binary name somehow? For example: pip-sync --python-executable=python3.

Great idea, I can try using shutil.which for resolving
https://docs.python.org/3/library/shutil.html?highlight=.which#shutil.which

Looks like it works well for aliases and full paths, so we can always use it for resolving:

In [1]: import shutil

In [2]: import sys

In [3]: shutil.which('python')
Out[3]: 'D:\\Envs\\test\\python.EXE'

In [4]: shutil.which(sys.executable)
Out[4]: 'd:\\envs\\test\\python.exe'

[atugushev]

I've caught a trace when a given python executable is not found:

$ pip-sync -a --python-executable=python3.9
Traceback (most recent call last):
  File "/Users/albert/Projects/pip-tools/.venv/bin/pip-sync", line 11, in <module>
    load_entry_point('pip-tools', 'console_scripts', 'pip-sync')()
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/albert/Projects/pip-tools/piptools/scripts/sync.py", line 126, in cli
    _validate_python_executable(python_executable)
  File "/Users/albert/Projects/pip-tools/piptools/scripts/sync.py", line 200, in _validate_python_executable
    log.error(msg, python_executable, pip_version, required_pip_specification)
  File "/Users/albert/Projects/pip-tools/piptools/logging.py", line 40, in error
    self.log(message, *args, **kwargs)
  File "/Users/albert/Projects/pip-tools/piptools/logging.py", line 24, in log
    click.secho(prefix + message, *args, **kwargs)
TypeError: secho() got multiple values for argument 'err'

[MaratFM]

My bad, did not re-run tests locally (on a side note tox -p all never produced all green result for me, even on master failing with coverage access denied errors).

I reverted logging message passing, was confused by the recommendation to delegate message formatting to the logging system (#1333 (comment)). Seems like pip-tools uses a custom logging wrapper that has incompatible method contracts.

[atugushev]

I reverted logging message passing, was confused by the recommendation to delegate message formatting to the logging system (#1333 (comment)). Seems like pip-tools uses a custom logging wrapper that has incompatible method contracts.

Thanks @MaratFM! Let's address that in a following up PR.

[atugushev]

There are too much of monkeypatches. Ideally, there should be only @mock.patch("piptools.sync.run").

[MaratFM]

refactored

[MaratFM]

@atugushev seems like with "# pragma: no branch" it stills shows partial coverage, so I reverted back to "# pragma: no cover"

[MaratFM]

@atugushev sorry for changing tests back and forth, I could not run tests for all these platforms locally and have to wait when someone will trigger the GitHub actions pipeline

[atugushev]

Thanks @MaratFM for your patience. Awesome work! 👍🏻