Improve release automation pipeline for core repository and docs

[glopesdev]

Right now releasing new versions of Bonsai is a multi-hour process with lots of manual steps. In Bonsai 2.8.4 we'd like to move towards automating this release process and improve the robustness of our continuous integration (CI) efforts in general.

At a high level, this means that the following will start being automated:

• Determining which packages have changed and will need to be published
• Building and publishing packages to NuGet.org
• Creating the installer and portable zip and publishing them to GitHub
• Publishing the new version number and download links to the Bonsai website
• Updating the docs website
• Other miscellaneous release chores like closing milestones and bumping version numbers

Additionally we hope to design this new infrastructure in a way that allows adding release automation to other Bonsai packages outside of the main repo. (EG: Bonsai.Arduino)

These changes also represent some other improvements to Bonsai in general:

• Bonsai is will be able to be built under the modern .NET SDK (first steps towards .NET 8)
• Continuous integration testing is now performed for Linux
• Source Link is enabled (this is a huge benefit to developers of Bonsai packages, also means Bonsai packages and binaries are tagged with the specific Git revision they were built from)
• Deterministic builds
• General build system cleanup and fixes

[PathogenDavid]

@glopesdev Fleshed out the issue description.

---

As per our meeting after dev club, how do we want to handle releasing packages only temporarily changed for unstable builds and preview versions of Bonsai?

After writing all this up I think I'm much more firmly in the "do nothing, assume this won't happen" camp, but I said I'd leave a comment for you so here it is :)

Determining which packages to release is done by comparing the packages between the current revision and the revision of the most recent stable release of Bonsai.

This creates an odd edge case for unstable and preview releases though. Imagine you have the following sequence of events:

• A: Release Bonsai 1.0.0
• B: Modify Bonsai.Dsp
• C: Modify Bonsai.Shaders
• D: Release Bonsai 1.0.1-alpha0
• E: Modify Bonsai.Dsp
• F: Revert change to Bonsai.Shaders
• G: Modify Bonsai.Core
• H: Release Bonsai 1.0.1-alpha1
• I: Release Bonsai 1.0.1

(The "Release" items are releases created on GitHub. The others are separate commits.)

For this sequence of commits, the following packages are published for Bonsai.Shaders and Bonsai.Dsp :

Event	Bonsai.Shaders	Bonsai.Dsp
A	1.0.0	1.0.0
B		1.0.1-ci0
C	1.0.1-ci1	1.0.1-ci1
D	1.0.1-alpha0	1.0.1-alpha0
E	1.0.1-ci2	1.0.1-ci2
F		1.0.1-ci3
G		1.0.1-ci4
H		1.0.1-alpha1
I		1.0.1

(Full and -alpha preview releases would go to both NuGet.org and GitHub Packages. -ci previews only go to GitHub Packages)

Basically the issue here is that Bonsai.Shaders stops getting releases since from the perspective of release automation it hasn't changed after the change is reverted in F.

---

As you mentioned during our chat, this is not a super likely scenario to occur and maybe isn't worth handling at all.

For CI builds we could just always release every single package (IE: ignore the release manifest from the package diffing job.) Would be a bunch of extra noise and storage use for a situation that may never realistically happen though.

I can think of some other partial solutions, but none without major downsides.

Solving it for previews specifically wouldn't be that hard (IE: ensure Bonsai.Shaders 1.0.1-alpha1 is released) but I think it's beneficial that the release manifest matches between preview and full releases. (In fact this is a nice benefit of using the release manifest for normal CI runs too.)

The slightly awkward scenario of Bonsai.Shaders 1.0.1-alpha0 never resolving to a full Bonsai.Shaders 1.0.1 is a bit unfortunate, but I'm inclined to think it doesn't matter. (If this ever happened and was causing confusion you could just unlist the intermediate preview versions.)

As I mentioned earlier in this comment I'm now thinking this situation just isn't worth worrying about. In the odd chance that it comes up, we can unlist problematic packages or make a dummy change to push people back to the no-changes version.

[PathogenDavid]

@glopesdev The other unresolved question that came up in our meeting:

Do we want to always release every single package upon a major release of Bonsai? (IE: Release Bonsai.Osc, Bonsai.Shaders, etc on 3.0.0 even if nothing in them changed.)

As with the other situation, while writing this I think I've personally gone from a "maybe yes" to a "probably no" on this one.

A possible reason to do this is that even though the packages will be functionally identical, it might be nice to draw a line in the sand of "This is Bonsai 3.0".

On the flip side though, this might incorrectly imply these old packages are still being developed even when they aren't. (Bonsai.Osc for instance has not had a release since October 2022.)

I realized though that this might be a moot point for Bonsai 3.0.0 since the .NET migration will implicitly touch every package.

In fact a reason to not do this might be to help users identify which packages are not available in Bonsai 3.0.0 yet. (For example, maybe we discover issues updating Bonsai.Scripting.IronPython and decide to skip it for the initial Bonsai 3.0.0 release.)

[PathogenDavid]

For posterity: We decided these situations aren't worth worrying about and that it's easier to do nothing since doing something does more harm than good.