3 Key Changes to the Lotus Release Process

[rjan90]

Hey everyone! We want to share some updates with regards to the Lotus release process going forward.

3 Key Changes to the Lotus Release Process

1. Revised Versioning:

We are revising the versioning strategy of Lotus to adopt a more user-friendly approach for our users, moving away from the mandatory (even) and feature release (odd) versioning scheme that has been the norm for the past couple of years.

After the Lotus v1.28.0 release is out which brings the changes for the next network upgrade (nv23), we will move away from the even and odd versioning.

The format will still be MAJOR.MINOR.PATCH, but with some nuanced changes compared to previous ways of doing it:

• Increment MAJOR if there are major changes to Lotus (e.g if we re-architect).
• Increment MINOR whenever there is a network upgrade, API breaking change or non backwards-compatible feature enhancements.
• Increment PATCH whenever there are backwards-compatible bug fixes or feature enhancements.

The Lotus Release Flow markdown document will be updated to reflect these changes once Lotus v1.28.0 lands (tracking issue). Its advised to subscribe to status.filecoin.io to get updates about dates for network upgrades. Whether a Lotus release is optional or mandatory for network upgrades will be clearly highlighted in the release changelog.

2. Separate Lotus Client and Lotus-Miner releases:

Historically, the development and release cycle of the Lotus Client and Lotus-Miner have been closely intertwined, while they serve different audiences. As mentioned in this recent Filecoin blog post, FilOz is carrying on the maintenance and the development of Lotus Client, ref-fvm and builtin-actors with a focus on network, protocol and node operator needs. While the Curio Storage team is taking on lotus-miner & Boost maintenance with a focus on developing a set of software that Storage Providers can use, as well as building Curio (often referred to as Lotus-Miner-V2).

FilOz would like to support different users’ needs better by separating Lotus client releases (for node operators like API-providers, exchanges, SPs, apps, indexers, toolings) & SP software releases (for SPs) so users can better evaluate what, when and and whether they need to update their system.

The first step in this direction is that the Curio software (miner V2) will have a separate repo as its home (follow the #fil-curio-announcements channel in Slack for updates). Lotus-Miner (V1) will remain in the Lotus repo. FilOz will be conducting integration testing outside of the production lotus-miner code. This means having enough functionality to test new protocol features, etc. The Curio team will decide what new features and bug fixes to support in lotus-miner based on user needs.

3. Faster releases

Last year we took a step back, and introduced a slower release cadence (bi-monthly) to improve internal processes to make the quality of code, reviews and testing processes better. We feel we are now at a point where we would like to shift gears back again, and ship Lotus Client releases more often, as we have observed that a lot of our users, especially critical tooling builders, are waiting for features & enhancements to be shipped in a stable Lotus Client release. At minimum, we will ship a release every month.

We will continue to improve code quality, reviews, paying down tech debt and looking at deploying nightly deployments on master, etc. But we feel confident that we have a good basis to have more Lotus Client releases again.

Rollout of these changes:

• Revised versioning: After Lotus v1.28.0 / NV23 (tracking issue)
• Seperate Lotus Client and Lotus-Miner releases: Curio software estimate to move out early July. Seperation of releases after that.
• Faster Releases: Effective immediately.

[jennijuju]

Thanks for writing this up.
when do we plan to rollout these changes?

[rjan90]

The revised versioning:

After the Lotus v1.28.0 release is out which brings the changes for the next network upgrade (nv23), we will move away from the even and odd versioning.

Separate Lotus Client and Lotus-Miner releases:

• The Curio software moving to a seperate-repo is ETA in June.
  • Seperation of the Client and Lotus-Miner releases can land after the move.

Faster releases

• We will move to ship releases faster right away

I will add a rollout-section to the OP, so that its a bit more clear when each of the steps will happen.

[jennijuju]

Separate Lotus Client and Lotus-Miner releases:

does this have to be blocked by Curio? Would look into if we can ship a lotus client release soon after v1.27.3 before nv23 release

[rvagg]

Yes, we can start working on that independent of Curio, I think. I don't believe there's anything stopping that except that it'll depend on how much other nv23 work "we" are tied up because shipping those features is probably highest priority, right?

Ticket for it here for tracking: #12010

[jleni]

ship Lotus Client releases more often, as we have observed that a lot of our users, especially critical tooling builders, are waiting for features & enhancements to be shipped in a stable Lotus Client release. At minimum, we will ship a release every month.

Given that monthly will be a minimum... Could you provide more information about the average expectation? and how often exchanges and other key stakeholder will need to upgrade? In particular, what will be the upgrade schedule for calibration -> mainnet?

[rjan90]

Hey @jleni! It seems like there are couple questions in here. So I will just try to clarify those:

1. Does more frequent Lotus releases mean that there will be a change when it comes to network upgrade schedules?

• No, more frequent Lotus releases does not mean that there will be changes in the network upgrade schedule. Ideally we will continue to try to uphold a 2 weeks upgrade time between a Lotus release candidate and a network upgrade on the Calibration network. And 3 weeks upgrade time for a network upgrade on the Mainnet.

2. How often exchanges and key stakeholders will need to upgrade?

• It´s hard to say how often they have to upgrade! If they do not encounter any issues with the current release they are on, and there are no new releases with vulnerability patches, or a release for a network upgrades, then upgrading is unnecessary. The goal for faster releases (and also having client and miner releases seperated) is to be able to bring bug-fixes and features faster out to end-users that need them, compared to what we deliver today.

[BigLep]

These questions/answers have been moved into #12322

[BigLep]

@rjan90 and I connected verbally on this a bit today. I'm including my notes from the conversation.

@rjan90 is going to create a tracking issue for the "Revised Versioning" work so it can have a checklist of the various steps that will need to be taken (e.g., update Lotus Release Flow markdown document), and so it can start showing up on boards/backlogs as a post nv23 item.

The issue will also capture more of the motivation for the change to no longer need odd and even minor versions to signal mandatory vs. feature releases. My understanding is that since LOTUS_RELEASE_FLOW.md was first written ~3 years ago, Lotus' stability and release quality have both increased, which minimize the need for having a "mandatory even" minor version that can be easily patched. Instead, by default, a hotfix can be applied to the latest production release, even if it includes additional released-in-production features since the last network upgrade.

[BigLep]

Tracking issue for revised versioning is here: #12072 . I also updated the main post with this tracking issue link.

[BigLep]

I believe another area we need to be clear about is the scope of changes that will be included in a Lotus release that corresponds with a network upgrade.

My understanding is that in the old world, we would make the guarantee to users that they could get a version of Lotus that only included latest fully released/deploy-to-prod software plus the minimum extra code needed for the network upgrade. This was accomplished by creating the new "mandatory release" branch from the latest "feature release" branch and then merging FIP-related PRs into that new branch and/or cherrypicking the minimum required commits that had already made it to master.

Example:

nv22 era:
1.26.3 - latest mandatory release
1.27.1 - latest feature release

for nv23:
release/v1.28.0 is created from release/v1.27.1. This is a "mandatory release".
Completed nv23 FIP-related work is cherrypicked from master.
Actively-landing nv23 FIP-related work is merged into release/v1.28.0

There isn't going to be a "feature release" for nv23. There is just the "mandatory release" of v1.28.0.

Commits that are in master at this time that weren't included in v1.28.0 will be included in v1.29.0 which won't be a "mandatory release" but will instead be a "normal monthly release".

Let's project out then then to nv24. Lets say that happens in October or November and that nv23 (v.1.28.0) wraps up in August. That means we potentially have something like:

• September: v1.29.0 - "normal monthly release" based off master
• October: v1.30.0 - lets assume this is the version that would be used for nv24.

I think the key question to be clear about is: would v1.30.0 in this example be based off master or v1.29.x? I believe there is a push to base it off master for release/branching simplicity. This does mean that consumers who are forced to update because of nv24 will be getting additional code in v1.30.0 beyond what was previously released and what is required for the network upgrade.

I personally feel this is a good outcome to achieve, but I also want to know what we would need in our test coverage or development process so that we feel good about having commits that haven't baked in prod previously be included in a network upgrade related release.

I also think we need to document this clearly given it's a shift.

[BigLep]

The was some #fil-lotus-dev conversation on this in https://filecoinproject.slack.com/archives/CP50PPW2X/p1718923493254039

[BigLep]

We have incorporated relevant information here in #12322

[BigLep]

For visibility, another area that has shaken out in executing on #12072 is deprecating the releases branch. This has been pulled out to #12374. Feel free to comment/question there.