Add ability to append build number to version identifier at build time

[TheFriendlyCoder]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Currently, poetry assumes / forces you to define your product version number explicitly in the pyproject.toml file. This is fine for building releases but doesn't work well when building pre-releases since such builds typically share the same target version. To avoid name/version clashes between packages, poetry should have some way to append a build number to the final package version at build time.

When using setuptools this is typically handled by appending a suffix such as dev or alpha followed by some sort of unique identifier such as a timestamp or build number. I'd like to see something similar in poetry.

What I thought might work best is to add an optional suffix parameter to the poetry build command which, when provided, would be appended to the version identifier stored in the the toml file. So if the static version in the toml file were, say, "1.0.0" and you ran poetry build --version_sufix = ".dev$BUILD_ID" or something similar, then the generated package would be versioned as "1.0.0.dev1234" (where "1234" is the current build number as defined by the environment variable BUILD_ID).

[TheFriendlyCoder]

NOTE: this feature enhancement is one of but a few fundamental problems that are currently blocking my team from adopting poetry as a full package replacement tool for all of our projects. In the absence of this feature - or something similar - we can not facilitate cross-developer builds where each developer is working on the same project release but on different branches. Each build needs to be uniquely labelled / addressable otherwise the packages will clash with one another when published to the package repository.

All this to say, this is a relatively high priority improvement for us.

[TheFriendlyCoder]

I probably also should mention that I am willing to put some effort into implementing this feature and submitting a pull request if the project maintainers don't have enough time to implement this in the near future. I'd just like to make sure the feature makes sense to the community as a whole before putting in the effort.

[areeh]

We have to work entirely around development wheels right now, so this would be useful.

I'm curious about the choice of arbitrary suffix string, or if you wanted to default to a suffix with compatibility with the rest of the world: The .devN suffix agrees with pep440 and pip. What about poetry itself? Perhaps this part of the source is relevant.

[asodeur]

I'd think this is the most wanted use-case for #693. Once this has landed it should not be long until setuptools-scm/versioneer-style plugins arrive. There have been tons of requests in that direction #5, #185, #672, #1034 and probably a lot more people waiting for this on the side-lines.

[dsully]

Ideally having an option to pass the version to the build command. We maintain dynamic versions that are centrally stored. I'd want to pass that version along to the build and have it embedded in the resulting sdist and wheels.

[mtkennerly]

It's technically possible to do this by running poetry version $VER before poetry build, but you'd need a script that can pass the right version to the command. I actually wrote a pseudo-plugin for Poetry for this specific use case, if you'd like to try it. The implementation is a hack until #693 is available :p

[HitLuca]

For anyone interested in using poetry with some basic automatic versioning, check out this past PR that was proposed, but declined in favor of a future plugin-based component.

[SailingYYC]

Looking for something quite similar to be able to update the version dynamically at build, but without it being reflected inpyproject.toml.

[Yajing-Zhao]

NOTE: this feature enhancement is one of but a few fundamental problems that are currently blocking my team from adopting poetry as a full package replacement tool for all of our projects. In the absence of this feature - or something similar - we can not facilitate cross-developer builds where each developer is working on the same project release but on different branches. Each build needs to be uniquely labelled / addressable otherwise the packages will clash with one another when published to the package repository.

All this to say, this is a relatively high priority improvement for us.

Hi @TheFriendlyCoder, I had the same feature request and came across your post. I understand why this stops you from adopting poetry. I wonder if you can share your current solution to this problem?

Thanks!

[WhyNotHugo]

I wonder if you can share your current solution to this problem?

Personally, I use setuptools-scm, which generates a version at build-time based on git metadata.

[sinoroc]

This might need the "Plugins" label.

[reitzig]

Workaround:

poetry version "$(poetry version).dev${BUILD_ID}"
poetry build # or publish

Assuming you have a numeric, increasing build ID, of course. I don't. :(

[matthewarmand]

Personally, I use setuptools-scm, which generates a version at build-time based on git metadata.

This works excellently when using setup.py because that file doesn't even need to show the version, it's derived at build/install time. The problem with using this with poetry is that poetry DOES require version in the pyproject.toml, so I'm not sure whether there's currently a way for poetry and setuptools_scm to coexist in as nice a way as it worked in setup.py. Would be curious to hear some best practices for this

[luqasz]

Personally, I use setuptools-scm, which generates a version at build-time based on git metadata.

This works excellently when using setup.py because that file doesn't even need to show the version, it's derived at build/install time. The problem with using this with poetry is that poetry DOES require version in the pyproject.toml, so I'm not sure whether there's currently a way for poetry and setuptools_scm to coexist in as nice a way as it worked in setup.py. Would be curious to hear some best practices for this

Yep I confirm. There is no way to use setupttols-scm with poetry. version field has to be present in pyproject.toml

[WhyNotHugo]

Yep I confirm. There is no way to use setupttols-scm with poetry. version field has to be present in pyproject.toml

Yeah, I figured that for anything where the version number is relevant, poetry is a bad fit -- the version needs to somehow be encoded into the file itself, but I'm using git for version control (e.g.: so git-tag is what I use to define a version, and git describe to determine it).

Encoding the version is a file is not a workflow that I've managed to make work when using git (or any other VCS).

[PabloEmidio]

Is there any news around this issue?

Any other 517 build backend implementation that I used has a way to define dynamic metadata (not just version, which is the main one): flit, hatch, setuptools

Although I use poetry in all my new personal projects, now that I finally got my company to accept using the new build backend specification, I'd to recommend other tool to attend our build process.

[PabloEmidio]

This implementation along with #1959 would be a game change to poetry when speaking about versioning.

[PabloEmidio]

And just to add a new point of view, maybe it'd be better split at two issue, but considering dynamic versioning; the poetry version command should be able to alter the specified file.

[PabloEmidio]

related to #3332

[gh-andre]

Just wanted to add to the conversation that a build number added to the version requires such package to be rebuilt for a release.

That, if a build pipeline creates abc.1.2.3.dev.57-py3-none-any.whl and QA tested it, in order to build a releasable package, a new package with the actual version would have to be built, which could introduce untested behaviors because build environments and packaging may have changed.

Wheels actually do have the official place for build numbers, which is supposed to be in this format:

{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-format

Problem is, I found no build backend that can actually generate this. setuptools even has the options that are supposed to add a build number, but they are non-functional and can only be used to generate local versions, as defined here:

https://peps.python.org/pep-0440/#local-version-identifiers

I found a workaround with wheel, which allows to inject build numbers into existing wheels, but it requires wheel 0.40.0 or newer and it only allows an actual number and doesn't follow the binary package format spec, which allows something like 123.main. Here's a bit more on this in a setuptools discussion.

pypa/setuptools#3811 (reply in thread)

[nfnvsc]

What do you think about adding a new bump rule, for instance development with an optional argument <build_id> so that the use case could be something like this:

Current Version	Command	Updated Version
1.2.3a0	poetry version development	1.2.3a0dev0
1.2.3a0dev0	poetry version development	1.2.3a0dev1
1.2.3a0dev1	poetry version development 10	1.2.3a0dev10

[gh-andre]

@nfnvsc The version sequence isn't supposed to contain build identifiers, which are supposed to be added in the build process and go into its own spot in the package name.

This allows one to build a package, test it and release that same package without repackaging it to remove build number from the version in the package name.

[WhyNotHugo]

@gh-andre I don't agree on this, though I'll admit it's a matter of personal preference.

I use git for version control, so I first create a git-tag for a version/release and then build it. The version declared in the tag itself is an input to my build process.

TBH, I don't think the feature discussed in this ticket is what you're looking for. If you build a package, test it, and then specify its version, then you need to mutate the built package, which is something different entirely.

[mikekuzak]

I'm doing this for the build version:

# Auto increment minor build version
poetry version "$(poetry version -s).${BUILD_NR}"

Does the trick for us

Packaging ... 
Python 3.8.16
Poetry (version 1.4.0)
Bumping version from 0.0.1 to 0.0.1.146570

Installing dependencies from lock file
Package operations: 16 installs, 0 updates, 0 removals
  • Installing pycparser (2.21)
  • Installing certifi (2022.9.24)
  • Installing cffi (1.15.1)
  • Installing charset-normalizer (2.0.12)
  • Installing idna (3.4)
  • Installing six (1.16.0)
  • Installing urllib3 (1.26.15)
  • Installing cryptography (40.0.0)
  • Installing numpy (1.24.2)
  • Installing pykerberos (1.2.4)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.7.1)
  • Installing requests (2.27.1)
  • Installing cachetools (4.2.2)
  • Installing pandas (1.4.2)
  • Installing requests-kerberos (0.12.0)
Installing the current project: py-mytest (0.0.1.146570)
Building py-mytest (0.0.1.146570)
  - Building sdist
  - Built py-mytest-0.0.1.146570.tar.gz
  - Building wheel
  - Built py-mytest-0.0.1.146570-py3-none-any.whl
Uploading files to Repositiory ... 
Uploading distributions to 
https://nexus.internal.com/repository/pypi-mcm-hosted/
Uploading py-mytest-0.0.1.146570-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 740.9/740.9 kB • 00:00 • 15.8 MB/s

[gh-andre]

@WhyNotHugo That's not what build numbers are for, though. Also, tags are intended to associated a released package with the source, not start the new version before a build.

A version starts with the Product Owner defining a release schedule. It could be a date or recurring releases, like train releases. Version should never be defined by a build process. Once defined by the PO, version gets into the source, like pyproject.toml and becomes a pending version that matures towards the release and build numbers reflect the maturity of each artifact package, as it goes through tests. Finally, when tests are successful, you release the tested package, without changing its name, and only then you tag what was released.

The reason it's not a personal preference is that each of these steps provides some capability for all release stages/audience, such as CI, CD, QA and, finally, users. If any is skipped, you will have to compensate for it. For example, build numbers are no accepted in public repositories (CI repos are fine, like Artifactory), so in order to get rid of the build number from the package name, you would have to repackage, undoing some of the testing in the process.

[WhyNotHugo]

@gh-andre Out of the dozens of projects which I maintain (and the several other dozens to which I contribute), none of them have a "Product Owner" or a dedicated QA team. Heck, even poetry itself doesn't fit this specific workflow that you're describing.

I get it, you work in some corporate environment and you have strict rules and workflows that you must follow. Other people don't work for the same organisation that you, and have different workflows. You can complain that "they're doing it wrong" all you want, but the world is full of people who do things different than you do, and you'll never be able to change that.

What are you trying to prove here? If you're just going to start discussing that people here are "using their tools wrong" because it doesn't fit your workflow then just move along. You're wasting your time and you're wasting our time.

[gh-andre]

@WhyNotHugo

Out of the dozens of projects which I maintain (and the several other dozens to which I contribute), none of them have a "Product Owner" or a dedicated QA team.

This just means that you are the Product Owner in that you define what will be released and what will be postponed for later.

In my reply I was careful to choose QA and testing as separate steps, meaning that in many pipelines testing will be automated and people may not always be involved. Same logic applies nevertheless.

I get it, you work in some corporate environment

I maintain a dozen of Open Source projects, just not in this account. Some for many years. I used to follow some of what you describe, until I realized that if I'm not releasing the very package that came out of testing - automated or QA'ed, I'm not doing a good job.

What are you trying to prove here?

Nothing. Just sharing what I learned not to be the best way to release packages for those who is interested. You clearly are not, which is fine. I will leave this particular sub-thread alone.

[reitzig]

Once defined by the PO, version gets into the source, like pyproject.toml and becomes a pending version that matures towards the release and build numbers reflect the maturity of each artifact package, as it goes through tests. Finally, when tests are successful, you release the tested package, without changing its name, and only then you tag what was released.

High-velocity, trunk-developed projects may work very differently. They may not even assign versions.

I for one am in "tag the commit, let CI do the rest" camp like @WhyNotHugo. The tag build is reproducible enough, I don't need versions in manifests. Tools like Maven or poetry get in my way with such a workflow, and I mostly just ignore the manifest version, similar to what @mikekuzak writes. Others may choose to have their pipelines edit the manifests to set the version, which has always seemed way too complicated for little gain to me.

(Never mind that PEP440 versioning has apparently been cooked up in hell; I guess semver wasn't good enough for Python? 👀 )

[gh-andre]

@reitzig

High-velocity, trunk-developed projects may work very differently. They may not even assign versions.

There's always a planned version, even for train releases and the like, where you bump the version at the end of the window automatically. This is not the same as bumping version for a build, which is what many are doing.

I for one am in "tag the commit, let CI do the rest"

One builds packages towards a release, where each package gets closer and more mature to fit the release criteria. Tagging beforehand automatically means you tag it before any testing was done.

I guess semver wasn't good enough for Python?

Because Python has many features not covered by semantic versioning, such as the local version used by package maintainers to fix up upstream bugs or ABI compatibility, and a few others. Many other package managers use similar techniques, like rpm, deb, apk, etc.

Besides, semantic versioning does define build metadata, which is supposed to be used for tracking build maturity. Using version as a substitute for build numbers isn't something SemVer promotes.

It all comes down to whether you can release the exact package that went through all of your tests for a given version. If you can, then whatever approach you use is working. If you test and then need additional steps to produce a package, like building from a tag, it's not, really.

[couling]

@gh-andre

A version starts with the Product Owner defining a release schedule. It could be a date or recurring releases, like train releases. Version should never be defined by a build process. Once defined by the PO, version gets into the source, like pyproject.toml and becomes a pending version that matures towards the release and build numbers reflect the maturity of each artifact package, as it goes through tests. Finally, when tests are successful, you release the tested package, without changing its name, and only then you tag what was released.

This is an incredibly narrow view of how a release cycle "should" work, and doesn't fit the workflow of the majority of companies I've worked for. There's a lot of things wrong with the workflow you've described, but as others have pointed out: It's personal preference.

Where you are absolutely wrong is to claim that build numbers "never" have a place in version numbers. That's just not true. You might not have seen cases where build numbers are needed in version numbers, but examples exist.

One example is for some heavily legislated contexts (medicine, aviation, gambling, ...) and certain legal jurisdictions, there are laws in place requiring the thing that is released to be byte for byte identical as the thing that was tested and approved independently. Re-running a build rarely results in the same byte sequence if only because the build time often get's rolled into the bytes of the file.

Would you get on a plane if they told you the firmware was not byte for byte the same as the firmware which was tested?

That is, in order to be legally compliant, some companies need to be sure that the build they release is the same build they tested, even if they re-ran the build several times for the same version of code in git.

It all comes down to whether you can release the exact package that went through all of your tests for a given version.

Ironically enough, that is the exact reason you have your build tag in the version number. This way, when a user tells you which version they are on, you can be 100% sure which build they are on, not just which version of source code.

[gh-andre]

@couling You're clearly misunderstanding pretty much all of what I described. By a mile, I might add. It's the other way around - deployments that need to know they are running the exact build that was tested when the package was released, identified by the build number and without repackaging, will benefit the most from this.

Sounds like you never debugged crash dumps or linked the source to the binaries in any way. A build number is routinely tacked into the version for a released package or binaries - take any packages with debug symbols of any kind, for an example, just not in the way you think it is. Build numbers/metadata don't come from the source, but from the build system.

[couling]

Sounds like you never debugged crash dumps or linked the source to the binaries in any way.

@gh-andre That's extremely rude, it's slanderous and frankly shows you are not interested in listening to others' experience.

What are you trying to prove here?

Nothing. Just sharing what I learned not to be the best way to release packages for those who is interested. You clearly are not, which is fine

You seem to be just trolling here and worse ...

The version sequence isn't supposed to contain build identifiers, which are supposed to be added in the build process and go into its own spot in the package name.

This allows one to build a package, test it and release that same package without repackaging it to remove build number from the version in the package name.

Package names are embedded in Python packages (described here) and any mismatch between package names makes them unrelated packages, so adding the build identifier to the package name would absolutely force you to repackage prior to release, not prevent it as you claim. What you described there couldn't possibly work.

[gh-andre]

@couling

All I'm saying is that one should release the very package they tested, without repackaging or rebuilding it from a tag. Not sure why this causes this reaction from a few participants in this discussion.

The rest is just examples, where tracking build numbers against released packages is very important for things like aligning it with debug symbols, etc, which cannot be rebuilt to match the binaries.

That's extremely rude

You called my point of view "incredibly narrow", so against that my observation that you probably didn't match source against crash dumps is just that - an observation. I apologize if this came out as offensive - it was not my intent.

[clintonroy]

Could we please stick to technical issues, and not attack anyone else's intelligence please?

Different people and organisations do different things, lets leave it at that.

I've had some success using https://pypi.org/project/poetry-dynamic-versioning/ to override versions at build time, I expect that could be used to inject a build number.

[couling]

Apologies that was unusually heated of me.

@gh-andre I think you've mixed up between two very different concepts and consequently said several things which are technically untrue. You seem to have confused between:

• PEP 440 pre-release suffixes which do require you to rebuild an artefact prior to actual release
• Build injected version numbers

There's absolutely no connection between these two things, and this this feature request has nothing at all to do with PEP 440 numbering. There is absolutely no reason why a build injected version number cannot go out as a the final release number as long as it doesn't contain PEP 440 pre-release or dev-release suffixes.

Build injected version numbers can typically be supplied from three sources:

• git tags or other SCM features

• inputs to build pipelines.

  Eg: in github actions drafting a "release" can trigger the build pipeline passing the release number as a variable. While the github release is in draft, the artefact can be tested prior to publishing.

• build time augmentation a version number supplied from somewhere else.

  Eg adding a build number to make 1.9 into 1.9.56712. Again, it is safe to release the version number 1.9.56712 if that's what your workflow dictates.

---

@clintonroy I've been using poetry-dynamic-versioning and it's okay for now. Unfortunately it still has some glitches and I frequently find a poetry update will modify pyproject.toml to disable it.

At this stage I'd much prefer a slicker way to embed the version number in via the CI pipeline, rather than the git tag, so to my mind the feature request is still quite useful.

[reitzig]

FWIW, my current approach is something like this (Jenkins-style Groovy):

def isReleaseBuild = "${env.TAG_NAME}" ==~ /^v\d+\.\d+\.\d+((a|b|rc)\d+)?$/ // cf. PEP 440

// NB: poetry normalizes the version we give it.
//     So, to avoid accidents, we let it do that upfront.
def versionProposal
if (isReleaseBuild) {
    versionProposal = env.TAG_NAME - ~/v?/
} else {
    def poetryVersion = sh(script: 'poetry version -s', returnStdout: true).trim()
    versionProposal = "${poetryVersion}.dev+${env.GIT_COMMIT[0..7]}_${env.BUILD_NUMBER}"
}
toolVersion = sh(
    script: "poetry version -s '${versionProposal}'",
    returnStdout: true
).trim()

This gives me flag isReleaseBuild and a version for this build (I don't care about re-running release builds), so later in the build I can do:

sh  label: 'publish',
      script: """
          poetry version -s '${toolVersion}'
          poetry publish --build --repository '${POETRY_REPOSITORY_NAME}'
      """

This changes pyproject.toml in the build agent, but I never commit it. There, it says:

version = "0.0.0" 

[abn]

While exactly using the build tag semantics specified by PEP 440, you can now use the local version label at build time see #9064.

The proper fix here really is the ability to add "build tag" component in the wheel name as well as the wheel metadata. Note that iirc, this is not supported for sdists.

Modifying dev/pre/post/rc at build time is not something that we should really support.

[abn]

According to PEP 440 the local version identifier was added to allow for build tags into the versioning of binary distributions.

Added the “local version identifier” and “local version label” concepts to allow system integrators to indicate patched builds in a way that is supported by the upstream tools, as well as to allow the incorporation of build tags into the versioning of binary distributions.

And according to PyPA Binary distribution format documentation

Build numbers are not a part of the distribution version and thus are difficult to reference externally, especially so outside the Python ecosystem of tools and standards. A common case where a distribution would need to referenced externally is when resolving a security vulnerability.

Due to this limitation, new distributions which need to be referenced externally should not use build numbers when building the new distribution. Instead a new distribution version should be created for such cases.

Considering all of the above, a local version tag is the right approach for this as opposed to a build tag.

This means #9064 resolves this issue and we will not support the use of build tags as it no longer recommended.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.