Support dynamic version generation, including from git tags and with git commit hashes

[lithomas1]

meson-python should provide a way to use the __version__ attribute of a package as its version instead of using the version from meson.

EDIT: for an extended description of this feature request, see https://github.com/FFY00/meson-python/issues/159#issuecomment-1266576327

[rgommers]

Thanks @lithomas1. xref pandas-dev/pandas#47081 (comment) for more context.

There are multiple things folks want to do with version names in wheel files, see e.g. versioneer, setuptools_scm and SciPy's custom code to produce a version. It doesn't seem like a good idea to support all that directly in meson_python. Allowing to use the __version__ attribute string seems like a way to avoid that.

One issue with __version__ is that it will not work for cross-compiling, because the produced wheel is in general not runnable. Also, I think it would require installing and running the wheel, which is too expensive. So a slight tweak on this proposal is using a hook of some sort to let the package author indicate to meson-python what the version is. Either as standalone runnable Python code, or by writing the version string to a file.

I am +1 on adding the feature, because this is something a lot of folks want - and want it enough to go through a lot of effort to get it.

[eli-schwartz]

I don't think that it's a problem for cross compiling. setuptools has long supported (via setup.cfg but not setup.py, IIRC) reading versions from an attribute of a source file, commonly __version__ -- it does so using:
https://github.com/pypa/setuptools/blob/646c71f9af7e9ba6677203acde4ccd0a478dfbf8/setuptools/config/expand.py#L62-L86

The tl;dr is that it takes foo.bar.baz and chops off the last bit to treat as a module attr, then uses the rest to find a suitable .py file which it parses via the ast module, searching for an assignment to that attr and parsing its value with ast.literal_eval. This is explicitly designed to do no arbitrary code execution at all, and only supports simple statements that can be literal_eval'ed, fortunately that is all that stuff like versioneer or setuptools_scm usually do.

Setuptools actually does fall back to directly importing the file if literal_eval fails. That's not precisely required, and obviously also has cross-compile concerns.

[rgommers]

Thanks @eli-schwartz, interesting, I didn't know setuptools had such a feature. I gave it a spin, it's a little fiddly:

>>> spec = _find_spec('scipy', None)
>>> spec
ModuleSpec(name='scipy', loader=<_frozen_importlib_external.SourceFileLoader object at 0x10622d5b0>, origin='/Users/rgommers/code/scipy/build-install/lib/python3.9/site-packages/scipy/__init__.py', submodule_search_locations=['/Users/rgommers/code/scipy/build-install/lib/python3.9/site-packages/scipy'])
>>> StaticModule('scipy', spec)
<__main__.StaticModule object at 0x1077a1fa0>
>>> StaticModule('scipy', spec).__version__
Traceback (most recent call last):
  Cell In [3], line 25 in __getattr__
    return next(
StopIteration

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  Cell In [20], line 1
    StaticModule('scipy', spec).__version__
  Cell In [3], line 31 in __getattr__
    raise AttributeError(f"{self.name} has no attribute {attr}") from e
AttributeError: scipy has no attribute __version__

>>> import scipy
>>> scipy.__version__
'1.10.0.dev0+1964.ac815d4'
>>> type(scipy.__version__)
<class 'str'>

>>> spec = _find_spec('scipy.version', None)
>>> StaticModule('scipy-version', spec)
<__main__.StaticModule object at 0x105a0c4f0>
>>> StaticModule('scipy-version', spec).full_version
'1.10.0.dev0+1964.ac815d4'

Given that __version__ is coming from generated code, it's going to be in a separate file. So you can't actually look for __version__ in the main namespace, you need the generated file where the actual literal string is.

The setuptools code is probably robust by now, if a bit obscure. So we could adopt it (the literal_eval part only) - one advantage is that our users usually already have a literal string somewhere in the installed package. It can be private, and it doesn't have to be named __version__. The feature would then look like:

[tool.meson-python]
version = 'scipy.version.version'

[eli-schwartz]

Given that __version__ is coming from generated code, it's going to be in a separate file. So you can't actually look for __version__ in the main namespace, you need the generated file where the actual literal string is.

Yup, that's how the logic is specified -- it sees if it can literal_eval the attribute "without code execution", and literal_eval means it has to be a simple assignment of a primitive type, no functions or imports or operators or any form of logic building. Which is all that's needed here.

The setuptools code is probably robust by now, if a bit obscure. So we could adopt it (the literal_eval part only) - one advantage is that our users usually already have a literal string somewhere in the installed package. It can be private, and it doesn't have to be named __version__. The feature would then look like:

[tool.meson-python]
version = 'scipy.version.version'

Yup, that's what setuptools does too. Since this is private logic to the internal source code / builder routines (pyproject.toml's tool sections) it's perfectly reasonable to poke at undocumented private attributes, and name them whatever you like into the bargain.

Is it necessary to look at the generated file, or is the plan to just look for the updated version after constructing the primary wheel contents, but before generating the dist-info, and reading it from the staged wheel layout?

[rgommers]

Annoying complication: we do want the sdist and wheel filenames to match, and we cannot ship generated sources in the sdist nor can we run run_command/custom_target nor have an install directory that we can inspect for a version string.

I don't see a good solution to that to make the pyproject.toml-based solution above work. A custom hook is easier to invoke, but still runs into the problem that we cannot ship generated sources (because all we do is run meson dist --allow-dirty) - and the git repo to wheel path should yield the same version as the repo -> sdist -> wheel path.

[eli-schwartz]

and we cannot ship generated sources in the sdist nor can we run run_command/custom_target nor have an install directory that we can inspect for a version string.

You can, Meson allows you to do this via a meson.add_dist_script() and there are projects that use this to write out a version file into the source tree. e.g. .tarball-version.

Usually this is not also an installable file created by a build rule -- it's just a flag file that allows a version generator script to deduce the version as a fallback if it's not running in a git repo.

That being said, it's not precisely forbidden to use

if fs.exists('version.py')
    py.install_sources('version.py', subdir: 'scipy')
else
    version_py_gen = find_program('version-py-gen.py')
    custom_target(..., output: 'version.py',
        command: [version_py_gen, ...],
        install: true, install_dir: py.get_install_dir() / 'scipy')
    meson.add_dist_script(version_py_gen, ...)
endif

It's also a use case I'd like to find a better solution for in the long run, because there are valid cases to ship generated sources in a release tarball. This is one of them. manpages and other forms of documentation are another.

[FFY00]

Going this route is possible, but seems a little bit hard. AFAICT the goal here is to only change the package version in one place, @lithomas1 have you considered rewriting __version__ via Meson itself instead (using https://mesonbuild.com/Reference-manual_functions.html#custom_target)?

[rgommers]

@lithomas1 have you considered rewriting __version__ via Meson itself instead

How does that help in getting the version into the sdist/wheel name? That is the one missing piece here; Pandas, SciPy et al. already have ways of generating __version__ (and it indeed already uses custom_target in SciPy).

[rgommers]

AFAICT the goal here is to only change the package version in one place,

That's not the only possible goal, and not the most important one. With versioneer (or a direct git invocation), the git hash ends up in the version string. You cannot get this from Meson. So what happens is that a code-generated _version.py gets shipped for setuptools based builds. That is the kind of thing folks want. So you have a generated version string, which can change commit to commit. Hence you cannot put that in meson.build. Channeling a version string back into meson-python so it uses that version string for file names and metadata is the intent here.

[eli-schwartz]

Afaict the real goal when you get right down to it is that whats-the-version-please.py can be run in meson.build to define the project version, and it can be run to produce a _version.py installable file which is in fact installed, and the former is presumably sufficient for meson-python to figure out the version as well.

The sticking point is that whats-the-version-please.py tells meson what the version is, and it's less obvious how meson is supposed to tell it what the version is... and that's relevant when creating an sdist, where meson knows when producing the sdist, what the version is (because whats-the-version-please.py told it based on git) but after the sdist is created, whats-the-version-please.py has no way of knowing.

It is simply the most familiar thing, to people, to have _version.py (however it's created! it doesn't actually matter) to roundtrip that, so that it gets created from git if needed, but if there is no git, then it's expected to have been created already, and snapshot that version string so that whats-the-version-please.py can use that.

That's how several different tools already work, including versioneer.

[lithomas1]

Thanks all for the replies. To be honest, I don't really feel strongly about meson-python having to get the version from __version__. It was just something that popped quickly into my head as something simple, and guaranteed to be correct. I do agree that it probably would be a hassle to extract it, though, I hadn't thought about the cross-compiling case when I first came up with it.

Ideally, version info would come from meson itself. One way to do this would probably be through the introspection files(intro-install_plan.json). We can probably mark the version file with a specific install tag, and then configure meson-python to look for a given variable in the marked file.

Also, slightly unrelated, but some general pain points in general:

1. vcs_tag generated files should be installable ( the install/install_dir keywords are missing)
2. vcs_tag should probably take in multiple commands, to generate different components of the versioning. The output should probably be limited to one file, though, as it is now. For example, in pandas we need to generate the __version__ string, and the __git_version__ string. Since, vcs_tag can only generate one component, we have to use custom_target, which is hacky.

[FFY00]

How does that help in getting the version into the sdist/wheel name? That is the one missing piece here; Pandas, SciPy et al. already have ways of generating __version__ (and it indeed already uses custom_target in SciPy).

You can have it in meson.build.

Ideally, version info would come from meson itself. One way to do this would probably be through the introspection files(intro-install_plan.json). We can probably mark the version file with a specific install tag, and then configure meson-python to look for a given variable in the marked file.

We currently support using the version declared in the Meson project.

See https://meson-python.readthedocs.io/en/latest/usage/start.html#specifying-the-project-metadata

Example:
https://github.com/FFY00/meson-python/blob/main/pyproject.toml#L36-L38
https://github.com/FFY00/meson-python/blob/25b8d286b68952f6ddb5bfcfc596c5002e852ae4/meson.build#L1

That's not the only possible goal, and not the most important one. With versioneer (or a direct git invocation), the git hash ends up in the version string. You cannot get this from Meson. So what happens is that a code-generated _version.py gets shipped for setuptools based builds. That is the kind of thing folks want. So you have a generated version string, which can change commit to commit. Hence you cannot put that in meson.build. Channeling a version string back into meson-python so it uses that version string for file names and metadata is the intent here.

I think I am missing something here, because I fail to see what that has to do with loading the version from __version__ 😅

I do agree that generating the version from VCS is a relevant use-case though.

The sticking point is that whats-the-version-please.py tells meson what the version is, and it's less obvious how meson is supposed to tell it what the version is... and that's relevant when creating an sdist, where meson knows when producing the sdist, what the version is (because whats-the-version-please.py told it based on git) but after the sdist is created, whats-the-version-please.py has no way of knowing.

Is there any escape hatch for that? Like specifying the version in an env var or Meson option for example. If not, it could be added, and we can make sure meson-python sets that when invoking Meson.

[eli-schwartz]

My suggestion up above was that meson.add_dist_script() can be used to run whats-the-version-please.py and write its contents to a file such as .tarball-version.

Is there any escape hatch for that? Like specifying the version in an env var or Meson option for example. If not, it could be added, and we can make sure meson-python sets that when invoking Meson.

And how does meson-python know what the version is?

If it did know, then whats-the-version-please.py could use the same method.

Meson doesn't support env vars, but whats-the-version-please.py could check for one. There's precedent for that in tools similar to versioneer, for example. Versioneer specifically doesn't (afaik), but setuptools_scm has $SETUPTOOLS_SCM_PRETEND_VERSION.

However, this environment variable is typically set by a build script such as a PKGBUILD or *.ebuild or rpm *.spec, not by a pep517 build backend.

[rgommers]

I think I am missing something here, because I fail to see what that has to do with loading the version from __version__

__version__ is simply an example here of "the desired project version string generated in some custom way, for example by versioneer".

You can see what version strings are used in nightlies here: https://anaconda.org/scipy-wheels-nightly. For example:

• Pandas: pandas-1.6.0.dev0+260.g2ef8d147c5-cp310-cp310-macosx_11_0_arm64.whl
• NumPy: numpy-1.24.0.dev0+854.g6f25c6274-cp310-cp310-macosx_11_0_arm64.whl
• SciPy: scipy-1.10.0.dev0-cp39-cp39-macosx_11_0_arm64.whl (used to look just like NumPy and Pandas before the move to Meson)

The reasoning here is:

1. Projects want version strings that include for example the git commit hash, or commit count since the last tag
2. Meson does not support that (the git hash part of it is being discussed for a long time, see git version meson#688 - and that's not even enough, for versions to sort correctly in nightlies, it needs the commit count)
3. So, the way to get the desired version is to run some Python code as part of the build process, to query git and to generate a file containing a string with the version for that particular build
4. The purpose of this feature request is to get meson-python to understand that version string, and include it in sdist/wheel filenames and metadata

[rgommers]

Here is a more complete list of packages that support dynamic version generation:

• versioneer
• setuptools-vcs-version
• setuptools_scm
• versioningit
• miniver
• setuptools-git-versioning
• better-setuptools-git-version (related predecessors linked in its readme)

There are even more random ones floating around, plus some projects (like SciPy) have their homegrown code for it. That shows that building this into meson-python is not a good idea probably. This is fiddly, and an endless source of bugs and requests for supporting other version schemes etc.

@FFY00 also pointed me to this way of exporting metadata with git-archive as a useful thing: FFY00/trampolim@2c6f7c4.

Perhaps one of the above packages can be an optional dependency to make it simpler for end users. However, the main priorities here are:

1. to update our documentation to show how to include generated files into the sdist (see this comment above),
2. have a simple way for a package author to tell meson-python where to look for a dynamic version string, so it's included in sdist/wheel filenames and metadata in .dist-info

[WarrenWeckesser]

Is there a cookbook or recommended best practice for this now? I'm porting a project to meson + meson-python, and currently I have version duplicated in pyproject.toml and in the project() of the top-level meson.build file. At the moment, I don't need to add the git SHA to the version, so all I'm trying to do is get rid of the duplication.

[dnicolodi]

You don't need to define the version in meson.build if you don't need it in meson.build itself, namely if you don't use meson.project_version(). If you want to keep the version in meson.build you can declare the version field to be dynamic in pyproject.toml and meson-python will pick it from meson.build

[project]
name = 'foo'
dynamic = ['version']

[WarrenWeckesser]

You don't need to define the version in meson.build if you don't need it in meson.build itself...

🤦 Ah, of course. That's what I get for "copy-paste-edit" development!