-
Notifications
You must be signed in to change notification settings - Fork 876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for platform independent lockfile and SHA-pinning #2679
Comments
Poetry and PDM support this feature, I think that it is essential for managing any kind of project that will be developed/run on different platforms. |
Yes, we absolutely plan to support this. |
@charliermarsh Are you planning on creating a new format or do you plan to export to I would personally be very grateful if you would target |
Right now I am having to generate the dependencies for each environment (windows, linux, mac) on different machines when running Instead of platform independent lock file, I believe it could be simpler to just allow uv to receive the platform parameters and build the file to the specified platform. Something like
|
I think the suggestion by @inoa-jboliveira would be a great intermediate step to getting cross-platform support for lock files. I would like to suggest to using the first two parts of the target triplet (i.e That would yield:
|
Yeah we've considered / are considering such designs but we're unlikely to implement anything like that until we've made a holistic decision on how we want to handle cross-platform locking. |
For what it's worth, there's extensive discussion around similar ideas here: https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/1. |
(I believe we could support a |
Not only it should be much simpler to support a target environment (since you kinda do that right now selecting the current environment) but it does not restrict you from later on having multiple targets so files can be portable cross platform. In terms of adoption of uv, I believe it is better to provide the functionality and later improve it than to be a blocker for many people. I'm pushing hard to replace all my tooling stack with uv on a large project. |
a |
Yeah I may hack on it when I have some time. |
Hi all! It would be awesome if |
I've prototyped this here: #3111 |
The next version will support |
I'm not going to close this though since the broader task here remains. |
@charliermarsh Thank you, looks very nice! Is it going to be released soon? |
It will probably go out today. |
#3111 is awesome and makes getting a "good enough" cross-platform lock file trivial. Dinky little example script: import argparse
import os
import subprocess
import sys
import tempfile
from pathlib import Path
from packaging.markers import Marker
from packaging.requirements import Requirement
def parse_requirements_txt(req_file: str) -> list[str]:
def strip_comments(s: str) -> str:
try:
return s[: s.index("#")].strip()
except ValueError:
return s.strip()
entries = []
with open(req_file) as f:
for line in f:
entry = strip_comments(line)
if entry:
entries.append(entry)
return entries
def marker_to_uv(key: str, value: str) -> list[str]:
if key == "sys_platform":
if value == "linux":
uv_platform = "linux"
elif value == "darwin":
uv_platform = "macos"
elif value == "win32":
uv_platform = "windows"
else:
raise ValueError(f"Unknown sys_platform {value}")
return ["--python-platform", uv_platform]
if key == "python_version":
return ["--python-version", value]
raise ValueError(f"Cannot convert marker {key} to uv input")
Environment = dict[str, str]
def env_to_marker(environment: Environment) -> Marker:
return Marker(" and ".join(f"({k} == {repr(v)})" for k, v in environment.items()))
def cross_environment_lock(src_file: Path, environment_matrix: list[Environment]) -> str:
cmd = ["uv", "pip", "compile", "--python", sys.executable, "--no-header", "--no-annotate"]
with tempfile.TemporaryDirectory() as tmpdir:
joined: dict[Requirement, list[Environment]] = {}
for i, environment in enumerate(environment_matrix):
out_file = os.path.join(tmpdir, src_file.stem + "." + str(i))
env_cmd = cmd + [src_file, "--output-file", out_file]
for key, value in environment.items():
env_cmd.extend(marker_to_uv(key, value))
subprocess.check_call(env_cmd, stdout=subprocess.DEVNULL)
for r in parse_requirements_txt(out_file):
joined.setdefault(Requirement(r), []).append(environment)
common = [r for r, envs in joined.items() if len(envs) == len(environment_matrix)]
cross_environment = [
(r, envs) for r, envs in joined.items() if len(envs) != len(environment_matrix)
]
cross_environment.sort(key=lambda r: r[0].name)
output = [str(r) for r in common]
for req, environments in cross_environment:
req = Requirement(str(req)) # make a copy
joint_marker = Marker(" or ".join(f"({env_to_marker(env)})" for env in environments))
if req.marker is None:
req.marker = joint_marker
else:
# Note that uv currently doesn't preserve markers, so this branch is unlikely
# https://github.com/astral-sh/uv/issues/1429
req.marker._markers = [req.marker._markers, "and", joint_marker._markers]
output.append(str(req))
return "\n".join(output)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("src_file")
parser.add_argument("--output-file")
args = parser.parse_args()
output = cross_environment_lock(
Path(args.src_file),
[
{"sys_platform": "linux"},
{"sys_platform": "darwin", "python_version": "3.10"},
],
)
print(output)
if args.output_file:
with open(args.output_file, "w") as f:
f.write(output)
if __name__ == "__main__":
main() |
Oh wow, that's cool. So you do multiple resolutions, then merge the |
Yup, uv's fast enough that resolving multiple times is totally fine. Unsolicited thoughts on what I want from a lock file:
With the above:
The best way I see to get 4/4a is you probably want something more like "record the input requirements, use the lock file as a source of constraints, at install time re-resolve the input requirements using the lock file constraints". uv exposes almost all the building blocks you'd need, I think the only thing missing is a few missing features from constraint files (e.g. interactions between constraints and hashes, express an "or" constraint, lmk if details are useful) |
@hauntsaninja In addition to the above list of nice properties to have in a lock file I'd like to add:
|
That should all be supported in the PEP I have coming (just got back from parental leave, but the draft PEP is written and just waiting on me to write a PoC which requires adapting some pre-existing code; the draft was also sent to @charliermarsh on discuss.python.org privately in March). I will be posting to https://discuss.python.org/c/packaging/14 once the PEP is ready. |
For those following, #3347 is tracking the lock file work in progress. |
In It's conceptually similar to |
It's definitely experimental! And part of the thinking here (in addition to shipping something useful in the |
Since uv 0.3.0 stabilized |
Yes, thanks! |
uv lock has been stabilized this version, which is used for generating lockfiles. See astral-sh/uv#2679 (comment)
uv lock has been stabilized this version, which is used for generating lockfiles. See astral-sh/uv#2679 (comment)
Hey, just noticed README.md still has this section:
|
@danielgafni -- Where are you seeing that? Are you looking at an old version perhaps? |
Oh damn... I'm on my phone so can't really track it down but this must be the case! 😄 sorry for the false alarm |
re-raising astral-sh/rye#881
The text was updated successfully, but these errors were encountered: