[Proposal] Switch master to next release after M3

[iloveeclipse]

Looking in the past changes in our release workflow I see a major disruption in development efforts for everyone contributing to the project after M3. This basically "mutes" everyone for a month, PR's got forgotten, external contributors disappointed etc.

On top of that even project contributors are often merge PR's into master after M3 without following the "RC" rules, for various reasons, even failed checks don't prevent them doing that.

Additionally I see a big issue with "maintenance" branches, which aren't even compilable anymore because eclipse.org doesn't run maintenance builds. Most recent case is 4.26 release branch which can't be built today because it uses tycho snapshot 3.0.0 which is simply not available anymore.

The proposal would be: once M3 is declared, immediately:

1. Freeze master till following tasks are done
2. Start "maintenance" (or let it better name "release") branch, do not change version there
3. Move master & I Builds to the next release

After M3, contribution of RC1 & RC2 would be done from the "release" branch.

Benefits I see:

• no disruption in contribution process
• no one can "per mistake" merge PR into RC build (higher effort to push a PR on a different branch as "master")
• the "release" branch build process is at least tested few times

If there are no technical issues, I would propose to start doing that with 4.27 M3.

Thoughts?

[jonahgraham]

In general this sounds like a good idea as there aren't many changes to master between M3 and current branch creation date. (Although I wish there was an easier way to cherry-pick PRs to other branches in the GitHub flow to allow changes to go into master and then be cherry-picked to 4.27 branch)

Will there be I-builds for the 4.27 after master moves to 4.28?

[iloveeclipse]

Will there be I-builds for the 4.27 after master moves to 4.28?

No, only RC builds. The idea is to start working on next release almost immediately (once all the boring "release" branch work is done, new jobs created etc)

[jonahgraham]

Does that mean that the build is only run twice (RC1 + RC2) after the branch? If I commit a change to 4.27 branch that breaks integration the day after the branch, when do I find out it fails?

[iloveeclipse]

Good point. So may be one can simply trigger a build job if needed

[HannesWell]

Wouldn't it be possible to run I-builds for both versions for some time?
The I-build on the release/maintance branch should probably only run If there are changes and maybe at another time to reduce the load of the build Infra.

[Bananeweizen]

Jonah, independent of the remaining discussion: That's why I require our (company) developers to start bug fixing on the oldest branch a bug exists on. Merging the older branch into the next newer one repeatedly is trivial (and even automated in some version control systems). Cherry-picking backwards from newer branches is an antipattern IMO.

[akurtakov]

That would put extra burden on releng side to update to latest EMF, Orbit, ECF, backport fixes, regen all documentation bundles, put N&N into docs bundles and etc. be done twice - once in master and once in release branch. I am not ready to take more work on that side and I doubt others that did this work would be happy to do more.
I welcome the idea but it has to come as a coordinated effort to e.g. remove N&N from doc bundles and replace it with simple external link (requiring network access to read it should be fine imho), stop shipping boilerplate edits like eclipse-platform/eclipse.platform.releng@94aaa13 and so on.
Overall, it's a call for move in the right direction but I don't think we can do it for this cycle without compromising any of the deliverable details. It would be best if someone drives this as a goal for June release.
@sravanlakkimsetti is the guy that knows best all these "last minute" changes that are being done.

[laeubi]

I think one keypoint would be to have true matrix builds in jenkins, e.g. for Tycho we already doing it that way:

https://ci.eclipse.org/tycho/job/tycho-github/

As you can see some of the older branches do not build (because no one cares) but currently we have 4.x development and still 3.x (and previously 2.7.x), so performing a "quiet" period would be just create a branch, do a small adjustment of the Jenkins file (for deployment of snapshots), and then people can either develop on master or backport/fix things on older branches.

Also cherry picking is quite easy, just fetch the branch (e.g. tycho-3.x), create a new branch (e.g. backport-fix123) then select the commit on the master branch and choose "Cherry-Pick", then push as PR and select the correct target branch.

So I think the key-point here is automate stuff as much as possible e.g for P2 I have setup CI-Friendly-Versions, that now only require a single change in the pom to upgrade the whole project to the next dev version.

[jonahgraham]

Also cherry picking is quite easy

FWIW I was comparing to gerrit which has a button and all happens on the server side. Not sure if github has an equivalent, but I haven't found it.

[laeubi]

I have never tried that, but it might work to just open another PR for the commit, target another branch, and then use the rebase button on github. Github Desktop has an option for this directly: https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/managing-commits/cherry-picking-a-commit

But if its really an issue, I think Egit can provide even better support for this as rely on some web UI ... especially if there are conflicts to resolve.

[akurtakov]

CI-Friendly-versions would definetely be a huge simplification.

[HannesWell]

If there are still simple modifications to perform/commit, like version bumps, this could be done in a manually triggered GH workflow that creates a corresponding PR.
The main steps could be defined in a shared Workflow e.g. stored in the platform.releng.aggr repo that all other Eclipse tlp repos reference (we do ist similar with the freeze-period Check).
When referencing such Workflow a list of Plugins/Features whoose Version has to be bumped in that particular repo can be passed.

Then to prepare for the next release one Just has to kick of that workflow in each repo, wait for the build result and merge it.
If we are confident, such workflow could even commit directly on the master.

Additionally we could even try to perform the Changes in a Single workflow (e.g. in the releng.aggr repo) that creates PRs for all submodules.

[akurtakov]

I have to say that even opening the new stream is quite cumbersome process with all the different repos to update parent versions, touch features, update Jenkins jobs and etc.
Step one in simplification from my POV is reducing git repos, step 2 is strip documentation bundles from content that should live on the webpage. Only after completing these obvious parts I would be able to estimate what's left and how much more speedier things can happen.
Of course, I happily pass the torch to someone else to try driving it faster :)

[laeubi]

This basically "mutes" everyone for a month

It would be interesting how many regressions/bug are actually found/fixed in this "quiet-month" so probably one can just reduce the quiet period at all.... I must confess that I have had some things on the table for this release but now its already over again, so probably just M1, M2 (with longer cycles) and then just immediately enter RC with bug-fixes only... and then just release it with shorter delays.

I have to say that even opening the new stream is quite cumbersome process with all the different repos to update parent versions, touch features, update Jenkins jobs and etc.

But I think that's independent from when it is done?

[akurtakov]

It is independent when it's done but it contributes by a week of freezing for "opening" new stream.

[sravanlakkimsetti]

Here are the tasks that needs to be done when switching to new stream
On the current release

1. Create maintenance/release branch
2. Move the build scripts and jobs to use new branch. (Till this is done you'll have no new builds for current release)

On the new stream using master (will take more than 2 days)

1. Update build scripts to new version
2. update pom/product versions
3. Create new jobs for prereqs updates, parent pom
4. Create test jobs
5. Trigger new I builds.

The problems I used face for this work is

1. Will not be able to differentiate I builds in the download page.
2. After release we normally clean all the I-builds for the release. This was a pain.

We have done this in the past. But it was really painful. I used work on weekends and 24 hours straight to the first part going(moving I-builds to maintenance/release branch).

The second part normally takes about a week. I seriously don't want any one to go back to that state.

[laeubi]

@sravanlakkimsetti do you think we can delay this a bit? I'd like to finish the work for the automated workflows so probably we can try this out already for a few repos if possible!

[sravanlakkimsetti]

Since no one asked for new milestones we can delay it. but keep in mind we need to open new stream on March 2. that would be deadline for us.

[laeubi]

Thanks, I'll tr to finish the test ASAP and give feedback here!

[laeubi]

@sravanlakkimsetti all test are done and workflows are prepared so the Milestones can be created if desired anytime.

[SDawley]

@SDawley the source for main download page is on the server itself. After losing access to ssh connectivity we are working with what is left on the server. If needed we can take a backup of the website scripts in to our code base.

@mickaelistria I did try to use two different folders for I-builds when we used to build for two streams simultaneously during 4.8 till 4.11 or so. I could not write the code for main download site(PHP) at that time.

@laeubi milestones are created using jenkins job

Circling back to this, I can't see or run the create-milestones job. I actually was unaware that Github milestones existed or were being created (assuming you're talking about https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/milestones)

So if these are going to be used to trigger the updateRelease workflows we should probably make the milestone creation process more accessible. If you could create a groovy file for the create-milestones job and add it to JenkinsJobs/Releng that would be very helpful. I also don't see this job or git milestone creation documented anywhere in the wiki, and its definitely not in RELEASE.md, do if you could either update one of those places or walk me though it so I can update the process to include it that would be much appreciated.

Also RC1 was last week but it looks like 4.28 Milestones haven't been created yet, so I would be happy to do that for this release since it would help me document it better, but as I said I cannot see the job.

[laeubi]

@sravanlakkimsetti @iloveeclipse @akurtakov I have now prepared the following PRs:

• Create updateRelease.yml eclipse-equinox/p2#222
• Create updateRelease.yml eclipse-pde/eclipse.pde#483
• Create updateRelease.yml eclipse-jdt/eclipse.jdt.core#772
• Create updateRelease.yml eclipse-pde/eclipse.pde#483
• Create updateRelease.yml eclipse.platform#338
• Create updateRelease.yml eclipse.platform.ui#611

These will need a review (and then probably a merge if nothing bad is detected).

[laeubi]

I merged it for all repos with merge-power now ...

[laeubi]

Please document how to use that in some wiki.

I can think about two ways:

1. we enable the wiki for the releng repo (more wiki style)
2. we add a readme.md to the workflows folder and document the workflows there (docs more closer to the 'code')

[akurtakov]

I'm in favor of 2. as if 1. is done there would have to be readme in the workflows pointing to the wiki.

[iloveeclipse]

If one searches for help "how to create new SDK release" without knowing about hidden files under .github/workflows/updateRelease.yml it wouldn't be a real help :-).
Having them in a wiki under releng repo would surely help more.

[laeubi]

Okay then we should first make sure what do you like to have described... "how to create new SDK release" is probably somewhere already described (@akurtakov @sravanlakkimsetti ?) and this does not change much, only that probably one step described there (e.g. make sure you update all versions in all repositories) are now hopefully automated by a workflow.
So if one want to know more about the workflow itself it is probably better be described there where the workflow is located (as its likely a technical than an organizational concern) and the general description of the process has just a link to "here you can find that workflow that automates some steps)...

[mickaelistria]

If one searches for help "how to create new SDK release"

Where would they search? This could be searched on GitHub, on a broader web search engine, locally inside the IDE, or with grep... and there are chances that occurrences get found by all of them if inside the repos; less chances if it's an external source.
But I kind of agree here that the documentation here isn't "top-level" and that some other documentation such as https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/RELEASE.md should be top-level and reference the "smaller-grain" ones.
Note that a wiki is not necessarily to have documentation; and it can even be an anti-pattern as wiki tends to be more easily out of date, do not enforce a culture of review (which utlimately leads to better collaboration and quality), and are too isolated for their contributions to count in the stats (while contributions to documentation can be as important as contibutions to code). Sticking to such a RELEASE.md file in the repo gives good results already.

[SDawley]

I created the RELEASE.md for myself originally because the releng wiki (which is linked) is vastly out of date and it was easier for me to start from scratch documenting the current process than try to update what was there. I'm also used to looking for documents like README.md and CONTRIBUTE.md and it made sense to keep the documentation with the code.

If Christoph's workflows are being used to update versions for a new release, links to the workflows and their documentation can easily be added in the https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/RELEASE.md#ga-releases section.

[laeubi]

I have now added a github action to Tycho that automatically creates a backport to a branch if one adds a given label:
https://github.com/eclipse-tycho/tycho/blob/master/.github/workflows/backport.yml

So the workflow would be:

1. Create fix for master
2. Add the desired backport label ('backport-to-tycho-3.0.x' in this case) to the PR
3. If the PR is merged, an automatic backporting PR should be created (I could not test this yet!)

A similar workflow would also work the other way round

1. Create fix for some-other-branch
2. Add label 'backport-to-master' to the PR
3. If the PR is merged, an automatic backporting PR should be created (I could not test this yet!)

[merks]

The initial change on any particular bundle or feature at the start of a new stream generally requires a version increment which would very often be wrong in a backport. I don't see a way to handle that automatically though, so just and observation that such a PR will typically need to be modified manually...

[laeubi]

I don't mentioned its all great and better, but if you feel so its great of course! Most of the time things are just different, and I never talked about any automatic version updates whatever that was brought on the table by you.

I only liked a solution I'm currently evaluating that seems to bring back a missing feature (I hope this is an acceptable term ...) to back-port a change from one branch to the other where I think it works similar. Still it will not solve all problems of the world, but usually its good enough to target the 80% then trowing out the baby with the water.

Still we might can enhance such features (and that's actually something that is great with github actios compared to some static features of whatever tool) to perform additional automation to help with the goal ... Tycho already has all features on-board to decide if/how version increments are to be handled.

[merks]

I think the idea behind @mickaelistria comment is interesting too. If one did the version incrementing as a separate commit (i.e., the part that one doesn't want to backport was done separate), that would help address the problem as well. Of course amending a commit is also not difficult...

[mickaelistria]

Tycho already has all features on-board to decide if/how version increments are to be handled.

Tycho has most features to usually decide if/how versions increments are to be handled. There will always be cases where we want a different increment in version than the suggested automated one (bumping by +0.0.1 on a maintenance branch vs +0.0.100 on main branch, or bumping by +0.1.0 without API change but because we want to emphasize on some important behavior change and because plugin.xml are not covered by API analysis, not bumping the version for features...).
Tycho can and does definitely assist better with proposals of how to bump some versions, and will be correct most of the cases, but it still require a review and cannot be fully automated IMO, because there are always corner-cases where rules are not sufficient, or are not consistent, and we need human decision.

About assisting doing version bumps; does the GitHub action you've build work on a branch (so anyone sending a PR can get feedback from the action and maybe even a comment with suggestions to apply? Or is it "only" with master, and the actions creates a PR ?

[laeubi]

First the action is not created by me! As far as I understand (I have not intensively tested it!) it can work on any branch and just creates a new PR. Of course one can then fetch the PR and adjust it if needed/desired even using the Girhub web feature to edit a file. But one can quite simply try things out, just add the action to the private fork and do some PRs, one can even ask some other person to create a PR against that private fork to see what happens with /without write access to repo and so on.

For version changes, Tycho can decide if a bump is required (of any form) so if one assumes some backporting workflow one can decide if the bump is probabbly not required, one can even say for backporting don't use +100 but use +1 Tycho can also compute if new api is added and so on ... and as said if we can cover most cases it is still a win if we only need to intercept on very small cases.

I just wanted to remind that the idea was that we just work on master during the RC phase so:

1. At best there are no changes to be backported as everything works fine
2. If we find a bug it should most of the time be a small and local change, so one will actually never add/remove API or alike so most version bump problems simply become quite trivial, so one can in almost all cases assume that any version diff between master > RC is not wanted and instead just a micro bump +1 is desired. Thats something one can automate quite easy.

[laeubi]

Here is an example for an automated backport I triggered right now:

• [tycho-3.0.x] Filter out items that could not be mapped eclipse-tycho/tycho#2225 (comment)

[merks]

Another important consideration is setting the (API) baseline...

[laeubi]

May I ask if/how we can proceed here? Is anything needed from Tycho/releng script side here, e.g. I opened:

• Feed the next streams update site repository with the contents of M1 / M2 / RC1 / RC2 prior to release #884
• Having an RELEASE+1 I-Build repository early #885

is there any plan for this in this release or are there any technical difficulties we need to solve?

Beside this, I must confess that I noticed (again) that I still don't get these release plans:

• https://wiki.eclipse.org/Category:SimRel-2023-06

The schedule says

2023-06 M1	Friday, April 14, 2023	04/07 to 04/14

and we already got the reminder, for "no big changes" so here again effectively muting development for another one and a half week? For what is this good for? Can't we have a 2023-06-M1 branch on Monday and right away go ahead possibly fixing found bugs on that branch and then if required?

Given that many contributions are done in free-time it seems quite disturbing to having align with these fixed time-frames, e.g. next "two development weeks" (if I get the calendar right...) will span the easter-holidays where many people are away from their PC, the M2 will already have its quiet-periods...

So to summarize: What is left so we can enter a much more continuous development cycle than all these disturbing freeze times ...

[akurtakov]

M2 is already done without freeze time and I have proposed to do so for M1 and M3 builds a year or two ago but haven't met enough support so I stopped this battle.
IMHO, there is no value in "milestone week" as what happens is pretty much everyone just moves to other things and spends even less time on Eclipse SDK.
If you have the power for this fight please bring it to eclipse-dev, hopefully this time we (as a community) are ready to accept that things are changing and what was a working solution in other circumstances is not in the current ones.

[merks]

It would be good to read the details here

https://wiki.eclipse.org/SimRel/Simultaneous_Release_Plan

The Platform is at +0 and +0 projects contribute/provide their contents on the Friday 1week ahead of the scheduled milestone. This is annotated on the calendar.

So the Platform tries to build their contribution by Wednesday, and asks people to test that build for sign-off by Thursday so that the final build can be provided on Friday. It asks the team to stabilize and focus the quality (bug/regression fixes) during this one week. ( Note that projects like EMF and Orbit are effectively at -2 because of the platform's schedule where the build is needed by Wednesday two days ahead of the Platform due date.)

So the team is being asked to focus on quality for more of four days. But qualfiy, as @akurtakov suggests, seems to be a low priority for many. Of course you can continue to work on your fork's branches continuously and ignore the disturbing freeze times, which is what many do. I suppose an alternative is to fork a branch that will be used for the M1 build in all the repositories, to have builds that use those, and to fix any encountered severe problems in those. But that's a daunting task and who's doing all that work when the actual important focus is continuous development without disturbances. So an alternative would be to continuously "dump" whatever the current state is, whenever it's needed, and hope that quality is either not important or is already ensured as we do for M2...

[merks]

@akurtakov I wonder with whom you were fighting a battle? It seems to me that the Platform decides what it wants to do mostly as it sees fit. That's certainly what I do with EMF and Oomph, but then I know exactly what's changing and who's changing it, i.e., me. The Platform has a whole whack of people changing things, and all-too-often things are broken for a short while from day to day. Do you really feel there is no need whatsoever to try to control/manage arbitrary activity right before delivering a stable build for broad downstream consumption? If so, I'm really not sure who or what is stopping you. I don't think it's the Planning Council nor any SimRel rules. And I don't think most of the "whole whack of people" care either. So where is the battle field?

[akurtakov]

Note that this is not done for M2 already and I haven't seen it being worse than other milestones. Anyway, here is last time where I proposed to have lightweight M1(aka without freeze just like M2) https://www.eclipse.org/lists/eclipse-dev/msg11975.html . It was not accepted by rest of committers so I assume majority of people are actively following the milestone week testing and etc. plan. I have to admit that I never proposed it for general voting (PMC hasn't had the policy back then for that) as I seem to be extremely tired of this kind of discussions.

For me it doesn't matter what is the best/better theoretical solution to a problem (which no one would execute upon) but which one is the one that reflects reality and could work. Even if something is worse than what/how it used to be but this is what happens now I feel obligated to try to change rules so they reflect reality and actually point out to interested people that something deteriorate rather than keep policy/processes just "on paper".

[merks]

Reading it again, I think there were some pretty constructive ideas in there with many about how to make it easy to eat your own dog food every day and just a few expressing concerns about the impact on quality which is I think the only strong argument against. I think the extent to which we (as many of the developers as possible) eat our own dog food, we weaken the arguments about the latter...

For me, what I do is Help -> Perform Setup Tasks which runs both the p2 task and the targlet task, so it installs the latest I-Build and updates the target platform to the latest I-Build. It's fully automated and takes a few minutes; I can work on other things while it does that. Selecting all the repos and doing a pull is also a habit (as is having all the repos in my workspace so that I see everything.) I see that folks like @iloveeclipse seem to find problems the fastest and from the mailing list thread, I see he has his own approach for ensuring he's testing that latest, so this really can be made to work. I've certainly worked very hard to make as simple as possible and as automated as possible...

I'm not sure about changing the rules simply "to reflect reality." If the reality ends up being that we deliver what are effectively integration builds with integration-build quality 4 times a year, that reality might be a really vibrant fun playground for playing with really cool features, but might well lead to those features becoming less and less relevant because the community is less and less impressed by poor quality. I'm not saying I'm against changing the rules, I just question (in what's in general becoming a fact-free age) who is defining the reality that the rules should reflect? I think everyone agrees that we should keep the "freeze" time frame was short as possible, or even eliminate it entirely if we can composite by doing not only continuous development but also continuous testing. We should consider how best to do that to avoid degrading into a playground mode. This need not be a battle...

[akurtakov]

So my question is: Are committers effectively following the "milestone weeks" testing/reporting/etc. rules? If not we have rule that gains us nothing but discussions. Anyway, I'm stopping here on the topic :)

[mickaelistria]

I personally don't, never have either; and never will.
I'm very happy with using daily I-Builds, and I report issues to those whenever I see them and without waiting further. I find the current process doesn't really allow me to detect nor report more errors, while I see it sometimes prevents me from merging fixes at a given time; so I just merge them a few days after (as part as a bigger batch of pending changes from other contributors). So no clear win here for me. But I agree that if even a few contributors manage to detect and fix more bugs thanks to this process, then it's great and maybe worth keeping; I would just not bet on it.

[HannesWell]

I fully agree with Mickael and are quite in the same situation. I don't have the time to systematically test Eclipse during the stabilization week, but I can use the latest I-builds and report issues I encounter.

But I agree that if even a few contributors manage to detect and fix more bugs thanks to this process, then it's great and maybe worth keeping; I would just not bet on it.

The number of people that really do that would probably be interesting, so that it can be estimated if it is worth the work.

Nevertheless I also agree that we should not just release an integration build four times a year and therefore should have some means to stabilize the released artifacts and try to ensure good quality.

[laeubi]

@merks I already read the link today (again) but as said I don't fully get it, its all quite vague (e.g. milestone dates are at roughly 2-3-week intervals, dates are typically one week apart) and seem to require a lot of in deep knowledge (with all those +X that are described as "projects choose it" but on the other had then restricted by other concerns) and do describe what is happening, but it does not describe why it happens, e.g. lets say platform will not "deliver M1" (how do we do it?!?) what will happen?

And yes it is officially a week, *but the mail says (emphasis by me):

Monday

• Last day of development, (and even then, no "big changes").
• After Monday 18:00 ET only regression fixes, no feature work or unrelated fixes.

Tuesday

• All-day test pass. Nobody should develop or fix anything.

So effectively this start 2 days before M1 ...

About stabilize things I'm all for it, but is there really a team of people dedicated to these testings while M1 / .. that are doing this exactly at these days? I can only second @akurtakov that my feeling is that it is more "stop all work", and we discover/fix bugs more often at the regular "development weeks".

So ideally we would have:

1. announce "Monday next week is M1 freeze, please finish / stabilize your work"
2. then at that date branch from master, build, deploy, ... from that branch
3. development on master can immediately continued
4. when any issues are found port them from master to ...-M1 branch (or vice versa), any new features or "unrelated fixes" just stay at the master branch.

That's why I asked if we need anything from a technical POV, e.g. an automation that creates branches / jobs / ... so there is just a non moving ground where all these processes can run that build the stuff and maybe allow for the " only regression fixes" to get in (maybe discovered already during the usual development that's going on in parallel).

[HannesWell]

The proposed procedure sounds good to me. It would allow (almost) continuous development, while still allowing to stabilize milestones and release candidates. Since you have already established a a back-porting workflow in Tycho that works, most of the technical problems seem to be solved.

One question to me would be, if there should still be a minimal freeze period, maybe one day, to create the M-X/RC-X branches or if that should be done by a job, on the milestone-monday at 12pm.

[merks]

As a concrete example of "why" the +0... If the Platform builds M1 on Wednesday and the Platform depends on EMF (not only depends on, but redistributes EMF's content), then it's best and ideal that EMF's build happens before the Platform's build such that the Platform builds with and tests with EMF's contribution. Because the Platform builds on -2, though delivers on +0, EMF is effectively at -2. So the "why" for the ordering is to produce some type of build progression based on the dependencies. Of course given there are circular dependencies, and given that people just choose when they want to contribute and which versions of their dependencies they build and test against, this is all nice in principle, but in practice it's more of a free-for-all.

As for dedicated teams, no this is all just chaos and a free-for-all. Yes, one finds bugs all the time during development, but unfortunately I also see that regressions are introduced all the time during said development. Moreover, much more time is focused overall on cool new things than on testing and fixing already-reported bugs. Bugs are often reported, even by the Platform developers, but the developers/contributors all too often move on to the more interesting work. This is just human nature and isn't going to change...

My sense of the proposal is that it only works if some dedicated team will manage branching all the repos and manage the builds for all those branches. Maybe that team is also responsible for testing; or maybe no one is responsible for that and we just hope people notice bugs while they work on cool things and hope even harder that they will fix those bugs and not just report them. In this way the most people can continue to work on the really cool stuff. As long as I'm not on the dedicated team working on the not cool stuff, I'm 100% in favor of such an approach so I'm sure most people will favor this approach. I think if we properly automate the process to reduce the grunt burden, that this is a workable approach that has significant merit.

My biggest concerns is that if we continue with this model through RC1 and RC2, as planned, with virtually no one actually testing the quality of what we are about to deliver to millions of consumers, the quality of what we deliver to the consumers may degrade significantly. So I think in this discussion one should take into account "who is doing the uncool work" and what is the tradeoff of focusing on cool work versus uncool builds and quality testing in terms of what we ultimately deliver. The perception of the millions of people who consume our results must also remain of high priority. I thinks it's possible to achieve all these goals. E.g., if folks regularly update to the latest integration build, which is trivially simple with the Oomph setups, they would be eating their own dog food everyday which could only help improve quality by virtue of extensive testing because of every-day use.