feat: support building a release in a dockerised environment

[mdelapenya]

What is the problem this PR solves?

This PR support building a release in an idempotent manner, where all build dependencies (Go and ZIP at this moment) are already present in the builder machine.

While building Fleet Server in a CI worker, we noticed that ZIP was not installed, and instead of creating "pet" workers with all tools installed, I considered automating the build at the project side, so that the build system is self-content, alowing developers to generate builds in a consistent manner.

How does this PR solve the problem?

It adds three make goals:

1. Build a Docker image based in a Docker image for building the release (uses make release as ENTRYPOINT). See Dockerfile.build for content
2. Run an ephemeral container of the above builder image, mounting project's workspace into the Go path, so that the entrypoint is satisfied
3. Check that all supported platforms are created under build/distributions. A shell script has been contributed to verify the existence of those files (Ideally we could extend that file in the future for further checks).

Finally, a new stage has been added to the main pipeline to check for release binaries existence. It's worth to say that this stage adds a few minutes to the pipeline, so maybe we can make it configurable... or maybe we discover it's so important that we agree in having those extra minutes.

How to test this PR locally

make docker-release
make test-release

If the release test passes:

If the release test fails:

It will create all binaries under the build/distributions directory (as expected), and run the tests verifying they are created.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

[mergify]

This pull request does not have a backport label. Could you fix it @mdelapenya? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[mergify]

This pull request does not have a backport label. Could you fix it @mdelapenya? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-11T10:17:25.987+0000

• Duration: 8 min 5 sec

• Commit: b6e6df8

Test stats 🧪

Test	Results
Failed	0
Passed	221
Skipped	0
Total	221

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[kuisathaverat]

you can use the docker golang-crossbuild images is a full Go build environment, this is a snippet from the bolt cli tool, it used the golang-crossbuild Docker images to build the binaries, also has support for cross-build if you needeed.

GO_VERSION = 1.16.4
GO_WORKDIR = "/go/src/github.com/elastic/tools/oblt-cli"
UID = $$(id -u)
GUID = $$(id -g)

...

## @help:crossbuild:Runs 'make clean build' ussing the docker.elastic.co/beats-dev/golang-crossbuild docker image to cross compile. It requires the environment variable (e.g. ARCH="linux/amd64") .
.PHONY: crossbuild
crossbuild:
	set -eux; \
	case "$${ARCH}" in \
		"linux/amd64") \
		CROSSBUILD_FLAVOR="main" \
		;; \
		"windows/amd64") \
		CROSSBUILD_FLAVOR="main" \
		;; \
		"darwin/amd64") \
		CROSSBUILD_FLAVOR="darwin" \
		;; \
		"darwin/arm64") \
		CROSSBUILD_FLAVOR="darwin-arm64" \
		;; \
		"linux/arm64") \
		CROSSBUILD_FLAVOR="arm" \
		;; \
	esac; \
	docker run -it --rm \
		-v "$(CURDIR):$(GO_WORKDIR)" \
		-w "$(GO_WORKDIR)" \
		-e CGO_ENABLED=0 \
		-u "$(UID):$(GUID)" \
		-e "HOME=/tmp" \
		-e "GOPATH=/tmp" \
		"docker.elastic.co/beats-dev/golang-crossbuild:$(GO_VERSION)-$${CROSSBUILD_FLAVOR}-debian10" \
		--build-cmd "make clean build" -p "$${ARCH}";

## @help:package-ARCH:Create a tar.gz file with the binary for the ARCH passed (make package-linux-amd64).
package-%:
	$(MAKE) crossbuild ARCH="$$(echo $*|tr '-' '/')"
	ARCH="$*"; cd build && tar -czf "../packages/oblt-cli-$(VERSION)-$${ARCH}.tar.gz" oblt-cli*;

## @help:package:Create a tar.gz file with the binary.
.PHONY: package
package: clean-all
	$(MAKE) package-linux-amd64
	$(MAKE) package-windows-amd64
	$(MAKE) package-linux-arm64
	$(MAKE) package-darwin-arm64
	$(MAKE) package-darwin-amd64

[kuisathaverat]

zip is not installed on golang-crossbuild images but we can add it and we use the same Docker image to build Go everywhere

[mdelapenya]

@kuisathaverat I will use your approach and add the cross build images, previously adding ZIP there. Thanks!!

[ruflin]

LGTM

[mdelapenya]

you can use the docker golang-crossbuild images is a full Go build environment

@kuisathaverat if you agree, let's do this in a follow-up PR

[v1v]

Approved, though there is a question regarding the implementation in CI Pipeline

[v1v]

@Mergifyio backport 7.17

[mergify]

backport 7.17

❌ No backport have been created

• Backport to branch 7.17 failed: Git reported the following error:

fatal: couldn't find remote ref master