HTTP Git credentials in `pyproject.toml` are ignored during sync #7453

flbraun · 2024-09-17T10:15:30Z

I'm currently evaluating uv for our Python teams. So far I'm very impressed with the overall snappiness (compared to Pipenv), thank you very much for the good work so far. Unfortunately I ran into an issue when dealing with private packages from our Gitlab.
I don't have knowledge about uv's internal workings, but my guess is that the authentication credentials aren't stored properly at some point and are then missing for successive operations.

$ uv --version
uv 0.4.10
$ uname -sp
Linux x86_64

First, I'm adding the dependency to the project:

$ RUST_LOG=uv=trace uv add "git+https://user:[email protected]/group/project.git" --tag "v2024.8.6"
[...]
DEBUG Fetching source distribution from Git: https://user:[email protected]/group/project.git
TRACE Checking lock for `https://gitlab.mycompany.com/group/project` at `/tmp/.tmp5Co2Yg/git-v0/locks/924509151ecc8240`
DEBUG Acquired lock for `https://gitlab.mycompany.com/group/project`
DEBUG Updating Git source `https://user:[email protected]/group/project.git`
DEBUG Performing a Git fetch for: https://user:[email protected]/group/project.git
DEBUG reset /tmp/.tmp5Co2Yg/git-v0/checkouts/924509151ecc8240/9bf0934 to 9bf093425188c4feaa7f86a78569b2fbbf5ba1a0
DEBUG Released lock at `/tmp/.tmp5Co2Yg/git-v0/locks/924509151ecc8240`
TRACE Checking lock for `/tmp/.tmp5Co2Yg/built-wheels-v3/git/8742abde34b47f9d/9bf093425188c4fe` at `/tmp/.tmp5Co2Yg/built-wheels-v3/git/8742abde34b47f9d/9bf093425188c4fe/.lock`
DEBUG Acquired lock for `/tmp/.tmp5Co2Yg/built-wheels-v3/git/8742abde34b47f9d/9bf093425188c4fe`
DEBUG No static `pyproject.toml` available for: git+https://user:[email protected]/group/project.git (PyprojectToml(DynamicField("version")))
DEBUG No static `PKG-INFO` available for: git+https://user:[email protected]/group/project.git (MissingPkgInfo)
DEBUG No static `egg-info` available for: git+https://user:[email protected]/group/project.git (MissingEggInfo)
DEBUG Preparing metadata for: git+https://user:[email protected]/group/project.git
[...]
Prepared 97 packages in 6.90s
Installed 97 packages in 179ms
[...]

As you can clearly see in the logs the credentials are present and the operation finishes successfully.

The package appears in pyproject.toml and uv.lock; note that the stated credentials are missing:

# pyproject.toml
[project]
...
dependencies = [
    ...
    "package",
]

[tool.uv.sources]
bxlog = { git = "https://gitlab.mycompany.com/group/project.git", tag = "v2024.8.6" }

# uv.lock
...
[[package]]
name = "thisproject"
...
dependencies = [
    ...
    { name = "package" },
]

[package.metadata]
requires-dist = [
    ...
    { name = "package", git = "https://gitlab.mycompany.com/group/project.git?tag=v2024.8.6" },
]

...
[[package]]
name = "package"
version = "2024.8.6"
source = { git = "https://gitlab.mycompany.com/group/project.git?tag=v2024.8.6#020cae642886a1fce3528366402047d479621904" }

Now, let's simulate another developer checking the project out and installing the dependencies for the first time:

$ rm -rf .venv
$ RUST_LOG=uv=trace uv sync --no-cache
[...]
DEBUG Identified uncached requirement: package @ git+https://gitlab.mycompany.com/group/[email protected]
[...]
DEBUG Fetching source distribution from Git: https://gitlab.mycompany.com/group/package.git
[...]
TRACE Checking lock for `https://gitlab.mycompany.com/group/package` at `/tmp/.tmpcPhaTx/git-v0/locks/924509151ecc8240`
DEBUG Acquired lock for `https://gitlab.mycompany.com/group/package`
[...]
DEBUG Updating Git source `https://gitlab.mycompany.com/group/package.git`
DEBUG Performing a Git fetch for: https://gitlab.mycompany.com/group/package.git
[...]
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: package @ git+https://gitlab.mycompany.com/group/package.git@020cae642886a1fce3528366402047d479621904
  Caused by: Git operation failed
  Caused by: failed to clone into: /tmp/.tmpcPhaTx/git-v0/db/924509151ecc8240
  Caused by: failed to fetch commit `020cae642886a1fce3528366402047d479621904`
  Caused by: process didn't exit successfully: `git fetch --force --update-head-ok 'https://gitlab.mycompany.com/group/package.git' '+020cae642886a1fce3528366402047d479621904:refs/remotes/origin/HEAD'` (exit status: 128)
--- stderr
remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found or you don't have permission to view it.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Looking back at pyproject.toml and uv.lock that's an understandable response - the username:password tuple is nowhere to be seen, there simply aren't any credentials that uv could have included in the request.

Interestingly, manually adding the credentials back into the URL in uv.lock makes uv sync operate as expected, but after the next uv operation that modifies the lock file (e.g. uv lock --upgrade) the credentials are lost again, so that's not even a valid workaround.

The text was updated successfully, but these errors were encountered:

charliermarsh · 2024-09-17T12:59:11Z

We intentionally do not store any credentials in plaintext files by default. You should be able to re-add them to the pyproject.toml if you actually want them to be persisted in plaintext.

zanieb · 2024-09-17T13:11:45Z

It's generally dangerous to commit credentials to your project, is there a reason that you need to do that?

You might want to look into the Git credential store which we will use automatically if the credentials have been populated. Some similar notes on this exist in the Cargo documentation.

flbraun · 2024-09-17T13:33:01Z

@charliermarsh Re-adding the credentials to pyproject.toml will keep them there, but they will never be included in uv.lock which ultimately results in the error described above.

I'm fully aware of the security implications of plaintext secrets. In this case it is fine however, since the project is only used within the company and the credential is a Gitlab deploy token, not a personal account. I won't deny that there are better ways of handling this, but that's the way it is right now.

uv's documentation explicitly states support for HTTP authentication, so I'd expect that to work with basic uv workflows without any big surprises:

Authentication can come from the following sources, in order of precedence:
- The URL, e.g., https://<user>:<password>@<hostname>/...

charliermarsh · 2024-09-17T13:35:53Z

If you're seeing the above trace after re-adding the credentials to pyproject.toml, then that's a bug, but your summary didn't suggest that you tried that -- only that you re-added them to the uv.lock. Did you? uv should not require that the credentials are present in the uv.lock file.

flbraun · 2024-09-17T14:03:52Z

Sorry for the confusion, let me clear this up:

In my original post I just manually added it to uv.lock but not pyproject.toml. That worked until I ran uv lock --upgrade which removed the credentials from uv.lock, leading to the described error when running uv sync --no-cache.

After your response I tried adding them to both pyproject.toml and uv.lock. That also works, but when running uv lock --upgrade the credentials are kept in pyproject.toml but are removed from uv.lock.
With the credentials only being in pyproject.toml running uv sync --no-cache fails to fetch the package again.

If you say that having it only in pyproject.toml is the desired state, then yeah, I guess we have a bug here.

zanieb · 2024-09-17T14:33:01Z

We have test coverage for this exact situation — I'm surprised this isn't working for you?

uv/crates/uv/tests/lock.rs

Lines 5999 to 6030 in fe8880b

	let pyproject_toml = context.temp_dir.child("pyproject.toml");
	pyproject_toml.write_str(
	r#"
	[project]
	name = "foo"
	version = "0.1.0"
	requires-python = ">=3.12"
	dependencies = ["iniconfig"]

	[build-system]
	requires = ["setuptools>=42"]
	build-backend = "setuptools.build_meta"

	[tool.uv]
	index-url = "https://public:[email protected]/basic-auth/simple"
	"#,
	)?;

	// Installing from the lockfile should succeed when credentials are included via
	// `pyproject.toml`.
	uv_snapshot!(context.filters(), context.sync().arg("--frozen").arg("--reinstall").arg("--no-cache"), @r###"
	success: true
	exit_code: 0
	----- stdout -----

	----- stderr -----
	Prepared 2 packages in [TIME]
	Uninstalled 2 packages in [TIME]
	Installed 2 packages in [TIME]
	~ foo==0.1.0 (from file://[TEMP_DIR]/)
	~ iniconfig==2.0.0
	"###);

zanieb · 2024-09-17T14:38:02Z

Does it work with --frozen?

charliermarsh · 2024-09-17T14:39:19Z

I think that's different @zanieb -- that tests an index, but here the credentials are provided on the URL directly.

zanieb · 2024-09-17T14:42:43Z

Oh good point! We have test coverage for that too, but the test might be wrong #7463

charliermarsh · 2024-09-17T15:25:52Z

Let's call it a bug for Git credentials then. I'll take a look.

…7474) ## Summary When syncing a lockfile, we need to respect credentials defined in the `pyproject.toml`, even if they won't be used for resolution. Unfortunately, this includes credentials in `tool.uv.sources`, `tool.uv.dev-dependencies`, `project.dependencies`, and `project.optional-dependencies`. Closes #7453.

flbraun · 2024-09-18T12:33:01Z

Thank you guys for addressing this so quickly! I'm running 0.4.11 now and everything works as expected.

zanieb added the question Asking for clarification or support label Sep 17, 2024

zanieb added bug Something isn't working and removed question Asking for clarification or support labels Sep 17, 2024

charliermarsh self-assigned this Sep 17, 2024

zanieb changed the title ~~HTTP authentication credentials are forgotten~~ HTTP Git credentials in pyproject.toml are ignored during sync Sep 17, 2024

charliermarsh mentioned this issue Sep 17, 2024

Respect pyproject.toml credentials from user-provided requirements #7474

Merged

charliermarsh closed this as completed in #7474 Sep 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP Git credentials in `pyproject.toml` are ignored during sync #7453

HTTP Git credentials in `pyproject.toml` are ignored during sync #7453

flbraun commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

zanieb commented Sep 17, 2024

flbraun commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

flbraun commented Sep 17, 2024

zanieb commented Sep 17, 2024 •

edited

Loading

zanieb commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

zanieb commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

flbraun commented Sep 18, 2024

HTTP Git credentials in pyproject.toml are ignored during sync #7453

HTTP Git credentials in pyproject.toml are ignored during sync #7453

Comments

flbraun commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

zanieb commented Sep 17, 2024

flbraun commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

flbraun commented Sep 17, 2024

zanieb commented Sep 17, 2024 • edited Loading

zanieb commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

zanieb commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

flbraun commented Sep 18, 2024

HTTP Git credentials in `pyproject.toml` are ignored during sync #7453

HTTP Git credentials in `pyproject.toml` are ignored during sync #7453

zanieb commented Sep 17, 2024 •

edited

Loading