Define Elemental code/release pipeline

[davidcassany]

I believe it is not clear enough what are we releasing and how all the repositories we are managing do relate to each other.

For instance changes in Elemental-cli hit OBS, CI and Elemental-toolkit by different paths, keeping them in sync pure manual work. Specially elemental-toolkit it playing a weird role in or current setup and we tend to neglect keeping it up to date, which is not good, we gathered quite some nice tests on the base OS in there, we should take advantage of all the work done there.

Currently in OBS and on rancher/elemental CI we have something like:

elemental-operator
                  \
elemental-cli      \
              \     \
               \     \
                -------- on merge/repack/tag ----> Elemental Teal*
               
* includes part of elemental-toolkit in sources by an automation copying files from one to the other

While IMHO the natural way of doing it would be something like:

                                    Elemental-operator  
                                                       \
                                                        \
Elemental-cli ---> Elemental-toolkit ---> Elemental OS ---> Elemental Teal
                                                       \
                                                        \-> Elemental OS derivatives

Elemental OS would be an image based on the elemental-toolkit artifacts and Elemental Teal would include only the Elemental Operator on top of it plus any additional required configuration specific to Teal (mostly the framework/files folder form this repo).

The whole idea is to put back Elemental-Toolkit back into the picture so our process forces going through it. Also the Elemental OS image even if not officially released I believe it should exist so our Elemental Teal image is based on it as well as other Elemental related OSes could do so. In addition, I believe this also relevant work to expose to the community even it would not be strictly related to any rancher use case.

The real challenge is how to map this into OBS. Currenlty in OBS the package isv:Rancher:Elemental/elemental includes the files from rancher/elemental that were copied from rancher/elemental-toolkit, this was done to facilitate the build in OBS at a time, but IMHO feels like a short cut.

One option could be splitting the isv:Rancher:Elemental/elemental into two, so one pulling sources from rancher/elemental-toolkit and another one pulling form rancher/elemental. Probably other options could be explored. The whole problem is having a build (or something equivalent) of rancher/elemental-toolkit in OBs.

I believe we should try to get into a position where on merge/tag our repositories the code flow is close or similar to second diagram.

[kkaempf]

The current OBS was chosen because our primary focus is on Elemental Teal. 😉

[davidcassany]

I created an environment in my OBS home with a setup closer to what I'd like ideally. My idea is to build three different OBS spaces:

1. dev: Builds from main branch of our repositories
2. staging: Builds from latest tags of our repositories
3. stable: Builds from pinned tags of our repositories

Number 1 and 2 are fully automated, while 3. is pinned manually. The only differences between the three is the _service file setup: 1 and 2 have the scm source enabled and setup with a webhook; while 3. has it disabled and with a known revision set.

You can see an example of dev in home:dcassany:Elemental:dev. There is new elemental-toolkit package exists that grabs sources from rancher/elemental-toolkit (it mimics luet packages defined there), so there is no need to include rancher/elemental-toolkit sources as part of rancher/elemental sources, this also allows us to better tag and ping rancher/elemental-toolkit in a meaningful way.

So in short there are four sources that the base of our release, so a release is a set of four tags of the following repositories:

rancher/elemental-cli
rancher/elemental-toolkit
rancher/elemental-operator
rancher/elemental

[agracey]

We really need to keep older (numbered) versions of the stable builds as well. This means that we can deal with recoveries of systems that haven't upgraded yet

[davidcassany]

We really need to keep older (numbered) versions of the stable builds as well. This means that we can deal with recoveries of systems that haven't upgraded yet

This what maintenance does by default. So any old public release will not disappear from server on further maintenance releases.

[davidcassany]

What #400 rancher/elemental-operator#198 PRs are meant to provide in isv:Rancher:Elemental:Dev

• All packages including sources from any of the following repositories can be updated by simply retriggering services in OBS (trigger services button click):

  • rancher/elemental
  • rancher/elemental-toolkit
  • rancher/elemental-cli
  • rancher/elemental-operator

  That means it is easy to automate source updates based on github events

• Elemental ISOs are built in OBS with a simple Dockerfile and they are published with static links:

  • https://download.opensuse.org/repositories/isv:/Rancher:/Elemental:/Dev:/Teal53/media/iso/elemental-teal.x86_64-0.1.0-Build12.1.iso (built artifact, version and build values come from node-image builds)
  • https://download.opensuse.org/repositories/isv:/Rancher:/Elemental:/Dev:/Teal53/media/iso/elemental-teal.x86_64.iso (static link to the build artifact without including version it keeps static across version bumps)

• rancher/elemental repository no longer includes sources from rancher/elemental-toolkit.

What I don't like from this setup:

• Including vendor folder as sources in rancher/elemantal-operator and rancher/elemental-cli
• Helm chart repositories are hardcoded in OBS sources, hence the repository setup is not portable across projects, this is a problem for maintenance and submission requests
• RPM specs are not tight to github changes (affects rancher/elemental and rancher/elemental-toolkit)
• There is no versioning chain between elemental-tookit and elemental-cli in OBS as there is rancher/elemental-toolkit (aka elemental-toolkit contains a pinned version of elemental-cli), so github and OBS are not in sync on that regard.
• Some weird artifacts are added into the elemental-operator helm chart to enable the build using pure OBS services.
• Elemental Teal Dockerfile in OBS uses an RPM for the included files, otherwise I could not manage to find a way to set tags according to the SCM service detected version
• OBS Dockerfiles and github Dockerfiles differ (github builds use Luet while Luet will be completely unused in official Elemental Teal from OBS)

What is still missing or to be completed:

• Set the OBS vs github webhooks (requires PRs to be merged) all merged already
• tag_push event is not fully functional, already contacted with autobuild, seams a bug in there side.
• Helm chart release and maintenance flow in OBS #411
• Adapt our elemental-build-iso script to consume released ISOs #376
• Elemental versioning #402
• End to end tests using iso and elemental-operator from OBS builds
  • ci/e2e: add a test for elemental-operator upgrade #412
  • e2e: add option to select Rancher Manager version and Teal ISO to test #426
• Build OBS packages on PRs affecting its sources, at least to verify there is no build break (this is nice to have, I think it is a relatively low hanging fruit)

[kkaempf]

considered done. #402 and #411 are tracked separately