The documented .git_archival.txt causes git archives to change hash after first post-release commit is made

[mgorny]

The documentation suggests using the following content for the .git_archival.txt file:

node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
ref-names: $Format:%D$

This means that if a tag is created on top of the main branch, the file initially contains e.g.:

ref-names: HEAD -> main, tag: v1

but once another commit is added and main no longer corresponds to the tag, it changes to:

ref-names: tag: v1

This causes the git archive hash to change. This is a problem for distributions like Gentoo that are using git archives generated by GitHub. The initial archive that we fetch once the release is made changes once upstream makes new commits.

See e.g.: https://bugs.gentoo.org/895910, https://bugs.gentoo.org/895712

[RonnyPfannschmidt]

Thanks for the heads up

Is there a suggested way to get that data

[mgorny]

I don't really know. I didn't even know such a thing is possible before setuptools_scm implemented it ;-).

[RonnyPfannschmidt]

Based on the git docs it's impossible to opt out of the problem details

Is there any reason why you don't use the release artifacts from pypi?

[ionenwks]

Is there any reason why you don't use the release artifacts from pypi?

In a lot of cases it's because they're missing something, e.g. tests (Edit: otherwise Gentoo does prefer pypi tarballs). Ideally everyone would include them but it's not always so easy.

[RonnyPfannschmidt]

That's unfortunate, in particular since setuptools_scm typically enforces that all files go in

[mgorny]

Yes, it is. It's been last debated in https://discuss.python.org/t/should-sdists-include-docs-and-tests/14578, with no definite conclusion. Unfortunately, some people believe that "sdist is only for pip" and so including files that pip (or a PEP517 build system, more generally) doesn't use is a waste of space.

[webknjaz]

Unfortunately, some people believe that "sdist is only for pip" and so including files that pip (or a PEP517 build system, more generally) doesn't use is a waste of space.

@mgorny I've been pointing people at my action https://github.com/re-actors/checkout-python-sdist to try to bring better standard into the ecosystem FYI. Projects that would use this, would probably be more downstream-friendly, as a result.

[eli-schwartz]

Isn't the need for ref-names obsoleted by the availability of the describe-name I added to current versions of git (and which is specifically designed to be stable, at least if abbrev is used)?

Maybe the template recommendation can simply phase out the old and unreliable ref-names formatter?

[maresb]

I have the impression that describe-name is relatively new and not so widespread yet. Specifically, I've noticed often warnings about describe-name being unavailable. When I locally build the aesara package under Ubuntu 20.04 (git 2.39.2), I get

* Building wheel...
/tmp/build-env-u3aklikk/lib/python3.11/site-packages/setuptools_scm/git.py:295: UserWarning: git
archive did not support describe output

Thus I fear that in practice removing ref-names could break a lot of workflows.

[eli-schwartz]

When locally building from a git repository the primary data source should be git itself, not the .git_archival.txt file... I wonder why this is triggering a warning.

[maresb]

Yes, that's a very good question, and I'd like to look into that. But in any case, I'm still concerned about the prevalence of Git versions which don't support describe-name.

[eli-schwartz]

In theory it should only matter in cases where you are not building from a git checkout, and you're not building from a PyPI sdist, and you are only building from the output of running git archive.

The main situation where that happens is on servers such as github.com, gitlab.com, git.sr.ht, codeberg.org, etc. and those tend to roll out new git updates much more often than new Ubuntu LTSes.

It is a good question, how often that will cause issues for local workflows. But I'm hoping the answer might be "not very".

[mgorny]

I also find it hard to imagine a use case for using a git archive locally. I always thought the main purpose of this feature is to support people fetching archives generated by GitHub, etc.

[RonnyPfannschmidt]

It's definitely time to go down the road of slimmer data there

Just recently I experimented with using the apis better

The new way is definitely the way to go

It still may be helpful to have ref names if limit to Tags is possible for broken forges

But those need to come forward

[dvzrv]

We have been seeing this issue in quite a few upstreams when packaging for Arch Linux by now and it significantly increases contributor time spent on figuring out what is going on and communicate with upstreams each time the reproducibility of a package is broken.

I suggest to either entirely remove the problematic .git_archival.txt documentation or to at a minimum add a big warning leading to this ticket and pointing out that it will make auto-generated source tarballs (and seemingly also tags?) not reproducible.

Is there any reason why you don't use the release artifacts from pypi?

• VCS sources or auto-generated source tarballs are more transparent for auditing purposes.
• sdist tarballs are not really well defined and often miss the stuff we need (tests, licenses, etc.)
• sdist tarballs are created in various environments
• backporting patches for setup.py/setup.cfg in sdist tarballs is a terrible experience
• we are now specifically encouraging the use of upstream provided source tarballs/ VCS sources (https://rfc.archlinux.page/0020-sources-for-python-packaging/)

[LecrisUT]

It should be documented which git hosts support describe-name and if they do simply remove ref-names. For Github projects just get rid of it.

The downsides of ref-names is arguably major enough that the project should not use setuptools-scm if the git host does not support descibe-name by now.