Sign darwin builds with rcodesign to prevent SIGKILL issues

[gitperr]

We currently don't sign the darwin builds, this causes node exporter to not boot properly. This PR attempts to fix that.

Fixes: #2539, #2217

[gitperr]

@SuperQ I am trying to produce darwin-arm64 binary, but the pipeline does not produce it.

ls -la results after build stage commands are run:
https://app.circleci.com/pipelines/github/prometheus/node_exporter/3954/workflows/ee5cbe9d-15d8-478f-a2c3-c0083b1c8368/jobs/20614/parallel-runs/0/steps/0-106

I tried to look at promu code, as well as the config files that are being passed to it, everything seems to be good... But, these commands don't produce the binary:

promu crossbuild -v --parallelism $CIRCLE_NODE_TOTAL --parallelism-thread $CIRCLE_NODE_INDEX
promu --config .promu-cgo.yml crossbuild -v --parallelism $CIRCLE_NODE_TOTAL --parallelism-thread $CIRCLE_NODE_INDEX

It did not work locally either (built everything else but darwin-arm64). I ended up trying:

promu crossbuild --config=.promu-cgo.yml -p darwin

which does produce the darwin-arm64 build. But it should have worked earlier already and this should not be necessary.

What should we do?

[gitperr]

Did some more reading of pipeline as well as promu, seems like the binaries get built based on which index they are on in the platforms. Since darwin-arm64 is on index 1, we'd need the CIRCLE_NODE_INDEX to be set to 1 while CIRCLE_NODE_TOTAL is 3 because of this calculations that happen on the here on buildThread method.

CircleCI actually did successfully build and sign the darwin-arm64 binary on node number 1.

But the end pipeline still fails because we attempt to sign on all 3 nodes, but some of them don't have the arm build as per the config. Will need to move signing elsewhere.

[gitperr]

Finally managed to build a signed darwin-arm64.

One other problem popped up:

• If the binary is downloaded using Safari, it will not run, warning the user with this popup

this can be bypassed by allowing it through macOS security settings. Probably the better way is to actually feed a code signing key, which is maintained by the project, to our code signing process.

I tried to download the binary using curl, which completely bypassed the above security warning and ran as intended.

If it sounds good so far, I can look into adding the key as well, but that would require a maintainer to put it to repo secrets probably.

[SuperQ]

We usually stick the binaries in a tar.gz, would that be enough to avoid Safari warnings?

[gitperr]

I downloaded the binary on my linux machine and tarred it to .tar.gz. Then hosted it myself (python3 -m http.server), pulled with curl on the mac and untarred it, works as intended.

If I click and download it with Safari, it even downloads the .tar.gz file as .tar, and untarred version still gives that popup. Quite weird.

[gitperr]

I did some more research and if we want to sign it with a key that's maintained by the project, we will have to pay Apple a yearly fee of $99 (that fixes the safari warnings completely). If we do not want to go that path, this can be merged anyway, because it improves the current SIGKILL situation if the user pulls with curl (or likely wget as well) but safari click downloads will require extra step to "allow it to run".

Ansibles etc., will end up using these tools, so it should not be a problem in that regard.

[discordianfish]

I'd say let's merge this for now and discuss the $99 thing separately. Thanks a lot of this!

[SuperQ]

I would rather build the signing call functionality into https://github.com/prometheus/promu. This way it can be spread to all Prometheus components more easily.

[gitperr]

Waiting for prometheus/promu#284 to be finalized and merged, this should come after that. We should re-trigger the pipeline to see if everything passes as intended.

[rexagod]

@gitperr JFYI #2916 (comment) needs a "Fixes" in place of "Fix" to close the relevant PRs once this is in.

[gitperr]

Edited the message: fix -> fixes

[discordianfish]

Looks like the promu version isnt updated

[gitperr]

Yep, it is still waiting for Julien's review. But if someone else wants to take a second look and give it approval, that might work too.

Otherwise I do not know how to move it forward. Help is welcome.

[gitperr]

Bump, I'd still like to get this forward and can make changes if needed.