Improve and automate releases

[serathius]

As mentioned on v3.5.3 release issue it took me 4 hours to run the release script. It required a lot of desperate patches to run and I don't expect anyone sane to do the same.

Some of the issues encountered when running release script:

• Path from local machine included in binaries. Requires a lot of manual steps to setup an environment to avoid leaking.
• Release sources/execute scripts from different branches in hard to predict way. There are a lot of bugs present in scripts from old release branches and incompatible changes done.

Changes that I needed to do to release v3.5.3 and v3.5.4 #13979

Proposed improvements:

• Release scripts are tested in dry mode on main branch
• Release scripts are tested in dry mode on supported branches (release-3.5, release-3.4).
  • Release scripts are tested on release-3.5
  • Release scripts are tested on release-3.4
• Process of building docker images is reproducible as it builds artifacts within docker image.
• Merging commit runs release process that pushes development artifacts.
• Tagging git repo starts automatic release process that uploads (most/all artifacts).
• Release process qualifies the release artifacts (runs e2e and functional tests)

cc @ptabor @spzala @ahrtr

[vivekpatani]

Is there any interest in generating rpms as a part of the release process? @serathius @serathius

[mrueg]

One thing that might be worth to consider is using https://github.com/goreleaser/goreleaser to build binaries for github releases.

[serathius]

@vivekpatani No there is no interest in expanding the scope of releases as we still need a lot of work to make quality of the release process reasonable.

@mrueg Thanks for suggestion, the main challenge is untangling the scripts used in existing release process.

[ghost]

Took a look at one of the PR's and the checks look incomplete to me. The goal should probably be to ensure that is in a state where we can actually trust that what's merged is being tested against the supported platforms.

I'll spend my next free cycle seeing if I can use the github workers for actions to be viable. They support Windows, Mac, and Linux. For the extra architectures supported, I'll see if I can make it happen with qemu or something in github actions.

Feel free to add me as an assignee on this. Gladly willing to help.

[serathius]

Took a look at one of the PR's and the checks look incomplete to me. The goal should probably be to ensure that is in a state where we can actually trust that what's merged is being tested against the supported platforms.

What you are describing is qualifying the released artifacts, or in other words release pipeline should not build new artifacts, but reuse ones that went through testing. This is very good property of release pipeline to have, however we don't have any release pipeline at all.

I have updated list of proposed improvements in the top comment. They list improvements to release process as a set of milestones in order that we should work on them. @protosam Please let me know if it make sense, happy to get feedback. Before jumping into the problem let's agree what we want to achieve and in what order.

[ghost]

I think we're on the same page, but I threw some charts together to reduce any ambiguity.

If the workflow is aligned like below, releases should be easier because the code base is "tested". This is just a rough draft, I realize it's missing a lot of details. Hopefully it makes sense.

After getting the build environments worked out to work like this, if we can get tests added in, the following step is to expect any contributed code to not be merged until it passes through this.

Pull Request Flow

Something worth pointing I wrote this as a trust-everybody model. It's not wise to just arbitrarily allow PR's to execute code.

flowchart TD
    subgraph PR_GRAPH [Pull Request Flow]
        PR([Pull Request Created])
        AWAITING_REVIEW[New changes need to be reviewed]

        PR-->AWAITING_REVIEW

        NEEDS_IMPROVEMENT[Request for improvements]
        CHANGES_MADE[Changes are made]
        PUSH[Changes pushed to PR Branch]

        NEEDS_IMPROVEMENT-->CHANGES_MADE
        CHANGES_MADE-->PUSH
        PUSH-->AWAITING_REVIEW

        PR_GH_ACTIONS_TRIGGERED[[Github Actions are triggered]]
        PR_GH_ACTIONS_RESULTS{" "}
        
        
        AWAITING_REVIEW-->PR_GH_ACTIONS_TRIGGERED
        PR_GH_ACTIONS_TRIGGERED --> PR_GH_ACTIONS_RESULTS
        PR_GH_ACTIONS_RESULTS --> |Successful|CODE_REVIEW
        PR_GH_ACTIONS_RESULTS --> |Failed|NEEDS_IMPROVEMENT


        CODE_REVIEW[Peer Review Needed]
        CODE_REVIEW_DECISIONS{" "}
        CODE_REVIEW_DISCUSSION[Discussion/Resolution Loop]
    
        CODE_REVIEW-->CODE_REVIEW_DECISIONS

        CODE_REVIEW_DECISIONS-->|Needs Go Tests|NEEDS_IMPROVEMENT
        CODE_REVIEW_DECISIONS-->|Needs E2E Tests|NEEDS_IMPROVEMENT
        CODE_REVIEW_DECISIONS-->|Code Issues/Questions|CODE_REVIEW_DISCUSSION
        CODE_REVIEW_DISCUSSION-->CODE_REVIEW_DECISIONS

        APPROVE_MERGE[Approve/Merge]

        CODE_REVIEW_DECISIONS-->|contribution is acceptable|APPROVE_MERGE
        
    end

Loading

Github Actions

flowchart TD
    subgraph GITHUB_ACTIONS_AUTOMATION [Github Actions are triggered]
        GH_ACTIONS_TRIGGERED([Trigger Event])
        GH_ACTIONS_MERGED_RESULTS{{Results}}
        GH_ACTIONS_RESULTS_DECISION{" "}
        GH_ACTIONS_RESULTS_RELEASE_DECISION{" "}

        GH_ACTIONS_TRIGGERED-->BUILD_MATRIX[Build Matrix]

        PREPARE_EMULATION[[Prepare Emulated Env]]

        BUILD_MATRIX-->DO_ON_darwin_amd64["darwin_amd64 (runner)"]-->PLATFORM_TASKS
        BUILD_MATRIX-->DO_ON_darwin_arm64["darwin_arm64 (unsolved)"]-->PREPARE_EMULATION
        BUILD_MATRIX-->DO_ON_windows_amd64["windows_amd64 (runner)"]-->PLATFORM_TASKS
        BUILD_MATRIX-->DO_ON_linux_amd64["linux_amd64 (runner)"]-->PLATFORM_TASKS


        BUILD_MATRIX-->DO_ON_linux_arm64["linux_arm64 (qemu)"]-->PREPARE_EMULATION
        BUILD_MATRIX-->DO_ON_linux_ppc64le["linux_ppc64le (qemu)"]-->PREPARE_EMULATION
        BUILD_MATRIX-->DO_ON_linux_s390x["linux_s390x (qemu)"]-->PREPARE_EMULATION

        PREPARE_EMULATION-->PLATFORM_TASKS
        
        subgraph PLATFORM_TASKS [Platform]
            GH_ACTIONS_PLATFORM_TASKS([Start Tasks])

            GH_ACTIONS_PLATFORM_TASKS-->GH_ACTIONS_RUNS_TESTS
            GH_ACTIONS_PLATFORM_TASKS-->GH_ACTIONS_BUILDS_BINARIES


            GH_ACTIONS_RUNS_TESTS[[Runs Go tests]]
            GH_ACTIONS_BUILDS_BINARIES[[Build binaries]]
            GH_ACTIONS_RUNS_E2E_TESTS[[Run E2E tests]]    
            GH_ACTIONS_BUILDS_BINARIES-->GH_ACTIONS_RUNS_E2E_TESTS

            GH_ACTIONS_RUNS_E2E_TESTS-->GH_ACTIONS_MERGED_RESULTS
            GH_ACTIONS_BUILDS_BINARIES-->GH_ACTIONS_MERGED_RESULTS
            GH_ACTIONS_RUNS_TESTS-->GH_ACTIONS_MERGED_RESULTS
        end

        GH_ACTIONS_MERGED_RESULTS-->GH_ACTIONS_RESULTS_DECISION

        GH_ACTIONS_RESULTS_DECISION-->|Pull Request|RETURN_AUTOMATION_RESULTS[End]

        GH_ACTIONS_RESULTS_DECISION-->|Tagged Release|GH_ACTIONS_RESULTS_RELEASE_DECISION
        GH_ACTIONS_RESULTS_RELEASE_DECISION-->|All Successful|PACKAGE[Package Binaries]
        PACKAGE-->CREATE_RELEASE[Create Release]-->PACKAGE_TO_RELEASE[Push packages to release]

        GH_ACTIONS_RESULTS_RELEASE_DECISION-->|Any Failures|NOTIFY[Log Issue/Notifications]
        NOTIFY-->|Should have not failed|TROUBLESHOOT[Fix Issues]
        TROUBLESHOOT-->|Fixes will be apparent in PR|RETRY_RELEASE[Retry triggering release]
        RETRY_RELEASE-->|Is there a proceedural problem to resolve?|ROOTCAUSE[Perform Root Cause Analysis]
        ROOTCAUSE-->|Future proofing|CORRECT_PROCESSES[Adjust Processes]
    end

Loading

[ghost]

The hardest thing to solve for the build system is going to be darwin-arm64. Getting it to run on qemu-arm64 is not trivial. Suggestions/ideas are welcomed on how to handle that, because I don't foresee Github having m1/m2 runners anytime soon due to licensing.

In the meantime, I'll move on to implementing the skeleton for everything else with a darwin-arm64 placeholder during my next free cycle next weekend.

[dims]

could we try macstadium? https://www.macstadium.com/opensource they seem to have what we need

[ghost]

@dims thanks a ton, this actually unblocks me. The way I was thinking about it was "how do I emulate M1", but moving forward I'll just expect that we should setup a Github Runner for darwin-arm64 somewhere (be it that vendor or a piece of hardware provided by one of the maintaining orgs).

I can test on my own hardware for now and we can figure out where to set that up when we cross that bridge.

[serathius]

Please let's make sure to make slow iterative changes instead of trying to redesign everything at once. It might be a little slower but smaller changes will be easier to get merged and start benefiting early.

[serathius]

@protosam are you still working on this?

[ghost]

Unassigned myself because I don't know if I'll be able to dedicate more time to this until December.

My prior diagram show a process of building in emulators and then doing testing. This was very slow. It is better to compile using GOOS and GOARCH natively. Then take the precompile binaries to be tested on the desired platform.

For cross platform testing, I was looking at this action. Haven't done much testing Debian 11 (bullseye) + arm64. It seemed like a good starting point for automating cross-platform testing.

[dims]

cc @dims