Improve VERSION.TXT contents

[svenevs]

Relates: #14509, #17736 (comment)

With the availability of staging-packaging jobs for releases we can populate VERSION.TXT with $ENV{DRAKE_VERSION} (rather than / in addition to) the date / sha.

Additional considerations:

• Create VERSION.JSON that contains any metadata we want now (or in the future) such as the commit sha etc.
• Make VERSION.TXT have just X.Y.Z to make it easier to extract it from command line based tools?
• Currently debian staging has a problem (Add --version $ENV{DRAKE_VERSION} to debian staging jobs drake-ci#185): if the release manager triggers the staging build on a different date, the date for the archive will be different than the other artifacts (name and VERSION.TXT). As such the release playbook details that only the .deb should be used from the staging, not the .tar.gz.

[jwnimmer-tri]

It's OK if, as part of this work, we end up changing the format of what's inside VERSION.TXT.

[mwoehlke-kitware]

Does the file need to look different (i.e. have actually different structure) for source/nightly builds vs. releases?

Right now I'm thinking to have some magic string instead of the version number for non-releases. What should that string be, however? It might also be nice to have e.g. "nightly" for those builds, however that might need a new environment variable similar to DRAKE_VERSION.

[jwnimmer-tri]

Let's work the big picture here, taking it from the top down (starting from https://drake.mit.edu/installation.html).

I think the first step is to assemble an inventory / summary / table of:

• all of the packaging artifacts we currently produce (tgz, deb, whl, docker, ...?);
• which jobs each can each kind be created from (nightly, staging, experimental, ...?);
• what extrinsic version number conventions are used in each (e.g., pep440 wheel version, deb package version, docker tag, ...?);
• what within-the-archive version file(s) are being shipped (VERSION.TXT, ...?) and what number formats are they each using?

That will help facilitate choosing a uniform numbering policy across the whole matrix, at which point is will hopefully be obvious what VERSION.TXT should look like and what extra CI machinery we might need in support of that.

[svenevs]

Relevant locations:

https://github.com/RobotLocomotion/drake-ci/blob/d36a0f73ac9a01b26fec696470b6e70691149699/driver/configurations/bazel/step-create-package-archive.cmake#L55

The idea would be to use $ENV{DRAKE_VERSION} when the track is staging, IIUC. That's what the other artifacts do, VERSION.txt being the problem child for .tar.gz and consequently .deb (since .tar.gz just gets repackaged to .deb).

drake/tools/release_engineering/repack_deb.py

Lines 54 to 64 in f54895e

	version = archive.getmember('drake/share/doc/drake/VERSION.TXT')
	version_txt = archive.extractfile(version).read().decode('utf-8')
	version_mtime = version.mtime
	packages = archive.getmember(
	f'drake/share/drake/setup/packages-{codename}.txt')
	packages_txt = archive.extractfile(packages).read().decode('utf-8')
	version_tokens = version_txt.split()
	assert len(version_tokens) == 2, version_txt
	yyyymmddhhmmss, git_sha = version_tokens

The deb repackaging will also need to behave differently.

Possibly relevant:

https://github.com/RobotLocomotion/drake-ci/blob/d36a0f73ac9a01b26fec696470b6e70691149699/driver/configurations/bazel/step-create-debian-archive.cmake#L62

Those are the only locations I'm familiar with this being an issue.

[mwoehlke-kitware]

Format	Track	Name	Has VERSION.TXT
tar	release	X.Y.Z	share/doc/drake/VERSION.TXT
tar	nightly	YYYYMMDD	share/doc/drake/VERSION.TXT
tar	experimental	YYYYMMDDHHMMSS-SHA40	share/doc/drake/VERSION.TXT
deb	release	X.Y.Z-1	/opt/drake/share/doc/drake/VERSION.TXT
deb	nightly	0.0.YYYYMMDD-1	/opt/drake/share/doc/drake/VERSION.TXT
deb	experimental	0.0.YYYYMMDDHHMMSS-SHA40-1	/opt/drake/share/doc/drake/VERSION.TXT
whl	release	X.Y.Z	pydrake/doc/drake/VERSION.TXT
whl	nightly	0.0.YYYYMMDD	pydrake/doc/drake/VERSION.TXT
whl	experimental	0.0.YYYY.MM.DD.HH.MM.SS+gitSHA8	pydrake/doc/drake/VERSION.TXT
docker	release	X.Y.Z	/opt/drake/share/doc/drake/VERSION.TXT
docker	nightly	YYYYMMDD	/opt/drake/share/doc/drake/VERSION.TXT

I believe we don't upload from experimental, only from nightly. In any case, I wouldn't worry too much about distinguishing between an experimental CI build and a user's local build. My point, however, was that it might be nice if official nightly builds were somehow tagged different from users' builds.

All releases use the same X.Y.Z version number. For others, the version number may or may not be 0.0.<date>. I suppose one option is to ensure that's used universally. If so, user builds should probably use PEP440-style 0.0.<date>+experimental or such. Otherwise, my vote is for the literal string nightly and something else for local builds.

[jwnimmer-tri]

I believe we don't upload from experimental ...

Yes we do (example), and we absolutely need to add the status quo of those version numbers into this table.

For others, the version number may or may not be 0.0.date.

That's not helpful. I want the actual version number patterns for all current CI output. I want everyone in this discussion thread to know exactly which tracks use which formats, at the moment. We need to know the contents of VERSION.TXT, not just whether or not there is one. For example, the release wheel 1.25 has this content in VERSION.TXT: 20240116195437 HEAD.

As you say, we should unify the new policy but we can't do that until we actually understand the current policy. I don't think anyone on this thread actually understands it. That's why I asked for a detailed summary.

Please work to make a comprehensive inventory of all tracks, without any ambiguity. We should have build logs & artifacts available for all of them, which you can inspect. Feel free to launch experimental jobs if we're missing logs & artifacts for any of those.

I want to know both the contents (pattern) of VERSION.TXT for each row, as well as the extrinsic version number (i.e., what the filename looks like).

[mwoehlke-kitware]

That's not helpful. I want the actual version number patterns for all current CI output.

All of them are of the form @BUILD_TIMESTAMP@ @GIT_REVISION@. That's the template used to generate the file; we don't (currently) know how to generate contents that don't look like that. If a VERSION.TXT is present, that's what it contains.

AFAIK, the file is always <appropriate-path>/doc/drake/VERSION.TXT. I updated the prior table with "exact" paths (tarballs and wheels have relative paths). If there are other relevant file artifacts, a) I'm not aware of them, and b) they don't contain version in their path (e.g. there is no __version__.py).

[jwnimmer-tri]

In the table I'd also like to see the artifact filename patterns we're using, e.g., for the releases we have files like this:

drake-1.25.0-jammy.tar.gz
drake-1.25.0-mac.tar.gz
drake-dev_1.25.0-1_amd64-jammy.deb
drake-1.25.0-cp310-cp310-manylinux_2_31_x86_64.whl

(That's what I meant by "extrinsic" version number.)

For the tagged releases these filename are obvious, but for Nightly and Experimental my recollection is that they were somewhat less obvious.

All of them are of the form @BUILD_TIMESTAMP@ @GIT_REVISION@.

I see now. The only caveat is that sometimes GIT_REVISION is a 40-character sha, and sometimes it's just "HEAD".

[mwoehlke-kitware]

The only caveat is that sometimes GIT_REVISION is a 40-character sha, and sometimes it's just "HEAD".

Hmm... then that's a bug; HEAD is in no way useful information. GIT_REVISION should always be a full SHA.

[mwoehlke-kitware]

For the tagged releases these filename are obvious, but for Nightly and Experimental my recollection is that they were somewhat less obvious.

Table updated.

[jwnimmer-tri]

#18145 (comment)

[svenevs]

Claimed Debian rules on version numbers: https://www.debian.org/doc/manuals/debmake-doc/ch05.en.html#name-version

[jwnimmer-tri]

We seem to all agree that we want the extrinsic package versions (what we name the files / wheels / debs / etc) to have a uniform numbering scheme. (Modulo the trailing -1 required by Debian, which we can ignore). So the next action here for @mwoehlke-kitware is to choose what that numbering scheme should be.

@jwnimmer-tri's action is to decide the contents of the VERSION.TXT file.

[mwoehlke-kitware]

AFAICT, Debian versioning is compatible with PEP440 versioning (at least as we're using the latter).

I'm confident the experimental YYYY.MM.DD.HH.MM.SS versioning is because something didn't like 14-digit numbers. OTOH, I wonder if we really need all those dots... and we aren't using them in nightlies.

Therefore, I'd like to suggest that all official packages use either X.Y.Z (releases) or 0.0.YYYYMMDD[.HHMMSS+gitSHA8] (anything else). This means that Debian and Wheel nightlies are unchanged, while all other non-release artifacts will move to a new scheme. (Some churn is unavoidable if we want to move to a consistent naming.)

One caveat, however, is that experimental versions will be "newer" than nightlies. (OTOH, this is arguably correct, since in most cases they will actually be newer, and presumably these are not "visible" via normal package providers.)

(To be clear, Debian versions all have a -1 appended compared to other versions. This isn't changing.)

[jwnimmer-tri]

SGTM

One caveat, however, is that experimental versions will be "newer" than nightlies.

To make sure I understand -- experimental builds would sort newer than the nightly that preceded them, but older than nightlies built afterwards, right? That sounds like what we want.

For example this ordering (from oldest to newest):

0.0.20240208                     # this morning's nightly
0.0.20240208.124701+git12345678  # experimental build from today
0.0.20240209                     # tomorrow morning's nightly

[mwoehlke-kitware]

Experimental builds would sort newer than the nightly that preceded them, but older than nightlies built afterwards, right?

Typically, yes. However, if the nightly is built at, say, 0300, and someone builds an experimental at 0100, the experimental will be "newer" (in terms of sort order). Generally, though, if we're okay with experimentals being newer than some nightlies (and I take your comment to imply that we are), I think this is fine.

Under the previous system, Nightly 0.0.19990101 would be newer than Experimental 0.0.2199.12.32.05.42.11, i.e. all nightlies are newer than all experimentals. So long as we aren't trying to preserve that behavior (and there's certainly an argument that that's just plain wrong, anyway)...

[jwnimmer-tri]

Ah yes, there is a kind of a race condition after midnight but before the nightly build kicks off. Yes, I think it's fine to have that bit of a gap / race. SGTM.

[mwoehlke-kitware]

FYI, it turns out CI overwrites VERSION.txt; the logic is at least partly entangled with the file name logic. So it probably makes sense to make both changes at the same time.

[jwnimmer-tri]

From f2f: we all seem okay with keeping the git sha in the VERISON.TXT file. Given that, we might as well just not change the formatting at all -- we just put the version number into the file, followed by the git sha.

[mwoehlke-kitware]

I just realized... only updating VERSION.txt when the user first runs cmake means someone building locally could have a significant discrepancy between the contents thereof and when they actually last ran a build. We probably should change generation of VERSION.txt from configure-time to build-time.

[jwnimmer-tri]

One more TODO: update the release_playbook.

[mwoehlke-kitware]

I found a couple remaining tasks:

• Update release playbook for recent changes #21359
• Deprecate --version in repack_deb.py #21364
  • Don't pass --version to repack_deb drake-ci#285
    • Remove repack_deb --version #21378
• Remove Bazel packaging drake-ci#284