Refactor CI jobs to use goreleaser

[Ashmita152]

Our current CI is too complicated to understand.

Requirement - what kind of business use case are you trying to solve?

I am creating this issue to start discussion around CI jobs refactoring. From my perspective, I can see following problems in our current CI implementation:

1. We do most of the stuffs ourselves in shell scripts which build tools like goreleaser can do for us. With buildx coming into picture, it has become even more complicated.
2. We have one huge Makefile which keeps on growing.
3. Lack of documentation around build scripts.

Proposal - what do you suggest to solve the problem or improve the existing situation?

I would like to propose following solutions for the above problems:

1. Let goreleaser do the heavy lifting for all our binary build, docker build and docker push related tasks. @jpkrohling already tried it for v2 in Setup CI for v2 #2886 and it looked much cleaner.
2. Divide root Makefile into multiple Makefiles.
   • Add each jaeger component specific directives to each cmd/<app>/Makefile and include them in the root Makefile.
   • With goreleaser into play, lot of directives should get removed.
3. Document the flow of build jobs.

Let me know your thoughts.

[yurishkuro]

I don't have a strong opinion about goreleaser. What makes it better? From the description it seems a sizeable portion of it is dealing with pushing GH release & homebrew.

I am not sure what splitting the Makefile gives us. The amount of content per binary is actually minimal in it. I would rather deal with a single 600-lines Makefile (which isn't that large) than hunt around for the same information in multiple dirs.

I think the last round of refactoring for buildx actually consolidated a lot of repeated steps into smaller number of scripts, which is great. The build process is still, indeed, pretty complicated.

[Ashmita152]

The trigger point for opening this issue is #3204 It clearly shows our CI is quite rigid. It works perfectly well on daily basis PRs but when someone need to change something in those shell scripts, it requires quite an effort and requires complete knowledge since we have too much coupling among those scripts. Amount of conversation in #2611 and #2948 reflect that pretty well.

Regarding goreleaser, I can see a few benefit:

• I feel all those shell scripts can be represented into yaml, yaml achors and aliases keeps it well maintained without need for repetition (Eg. https://github.com/jpkrohling/jaeger/blob/64333374cc658e3f9448869e592e0ddac7628341/v2/.goreleaser.yaml does almost all of the stuff we do)
• We don't need to document the working of our CI jobs, goreleaser has great documentation. Someone who needs to change the CI job can just refer the documentation of goreleaser. With scripts, they need to find our which script is calling what and understand the flow.
• Right now we write shell scripts, wrap that into Makefile directives and then call those Makefile directives from GH Action. Goreleaser has its own GH Action (https://github.com/goreleaser/goreleaser-action) This will reduce size of the Makefile and shell scripts drastically.

[yurishkuro]

I feel all those shell scripts can be represented into yaml

To me, converting from Shell scripts to YAML would be a huge step back. Everyone knows how to read and write Shell, and how to debug it. YAML is always a bespoke DSL that people need to learn from scratch, and it is usually much harder to debug.

yaml achors and aliases keeps it well maintained without need for repetition

This is just as easily achieved by reusing shell scripts or shell functions. In the example you linked, I do not see that it is substantially more concise (but it is substantially more opaque, bespoke, and non-debuggable).

Is goreleaser going to create Dockerfiles for us? I assume not, so it's not removing that complexity. So which steps is goreleaser going to substitute? If we're only using it as a replacement for go build and docker build, then the complexity is not in the commands, but in the preparation of various parameters and their permutations, and I am not seeing how it would make it to have fewer parameters.

Maybe I am missing something.

[pavolloffay]

It works perfectly well on daily basis PRs but when someone need to change something in those shell scripts, it requires quite an effort and requires complete knowledge since we have too much coupling among those scripts.

The biggest issue from my point of view is that I cannot run the full build locally. For instance to build docker images a local docker registry is needed. I tried running the registry locally and I got into configuration issues with TLS between registry and docker push.

For productive development, I would like to be able to run all commands locally e.g. build binaries, docker images and run integration tests. I don't care too much if it requires more code in the makefile/scripts.

[pavolloffay]

Our CI definitions look fairly simple, however if we can further simplify building of binaries and images I am all for it (e.g. via goreleaser). At the same time I would like to be able to build and run everything locally without decrypting CI configs.