OTP macOS builds

[wojtekmach]

I'd like to propose setting up a central place where macOS binary builds can be downloaded from.

The main use cases are:

• asdf (and similar tools) - Currently, installing OTP using this tools requires building from scratch. Which requires additional steps and there have been problems and caveats over the years: https://github.com/asdf-vm/asdf-erlang?tab=readme-ov-file#osx.

• beamup

• erlef/setup-beam - definitely less of a priority given it's rather rare to test on macOS. Still, it would be very useful for cross-platform projects like Livebook and LiveView Native.

Existing Solutions

• brew install erlang - It uses dynamic linking which has tradeoffs. At the moment, latest OTP version is slightly outdated: https://github.com/Homebrew/homebrew-core/blob/37eae9b8c890ebad9a0ebde257894f72132a27e6/Formula/e/erlang.rb#L6.

• https://beammachine.cloud / https://github.com/doawoo/beam-machine - Latest OTP version is slightly outdated. No wxwidgets (and thus observer and some other apps) at the moment.

Worth mentioning that the Hex team maintains OTP builds for Ubuntu: https://github.com/hexpm/bob?tab=readme-ov-file#erlang-builds. This is what erlef/setup_beam uses on Linux. At the moment, Bob (an Elixir app) regularly checks the OTP repo for new releases and builds them. We'd like to eventually move from that process to using GitHub actions so there's less infrastructure to maintain for the Hex team, more visibility into the builds, and people can more easily contribute (they can test their changes in a fork, for example).

Dynamic vs Static Linking Tradeoffs

OTP depends on third party software like OpenSSL and wxWidgets and can be dynamically or statically linked to these.

Dynamic linking results in smaller binaries, a given version of, say, OpenSSL, is shared between other programs that need it. If there is a vulnerability and a new version out, we can do brew upgrade openssl and all programs are patched (so to speak). On the other hand, an automatically updated OpenSSL (say jumping from 1.1.1 to 3.0.0) could break OTP crypto app.

The biggest benefits of static linking is the things work out of the box without installing additional dependencies. Additionally, it's a rare use case in the grand scheme of things, but Livebook ships one click installers for macOS and Windows and we REQUIRE an OTP with statically linked OpenSSL.

The main focus here is for development so lack of easily upgradable OpenSSL is perhaps not a great concern.

The OTP team maintains binary releases for Windows and both OpenSSL and wxWidgets are statically linked.

Goals

• Regularly updated OTP builds.

• Statically linked OpenSSL and wxWidgets.

• Separate build for x64/arm64. Alternatively, a single universal build (using lipo(1)).

• Easy to contribute.

• Easy to programatically list available versions. This is needed by asdf list-all erlang and erlef/setup-beam version matching.

Non-Goals

• Don't host older OTP versions. I'd only support OTP 27+.

Proposed Solution

Create https://github.com/erlef/otp_builds.

The repository would contain three things:

1. A .github/workflows/build.yml that builds OTP. The build would be triggered using workflow_dispatch event. This way we can trigger the build manually from GitHub UI or using GitHub REST API. After the build is built, it is uploaded to GitHub releases mentioned below. One nice thing is the build log would be public, if something went wrong, anyone could jump in and investigate.

2. Git tags and GitHub releases that follow OTP releases, e.g. OTP-27.0, OTP-27.0.1, etc. We could additionally have tags for master, maint, and maint-<release> which would be built, say, nightly. This is slightly unusual in that all git tags would point to a commit in this repo (with mostly just build scripts) as opposed to the source code the build was made from. On the other hand, using GitHub releases is very practical. They are discoverable in the UI, the release assets are easily downloaded (https://github.com/{user}/{repo}/releases/download/{tag}/{file}) and managed (REST API, gh CLI, etc) programatically. If builds are downloaded for GitHub Actions CI runs, it would make sense downloading from GitHub releases would be faster and more reliable than from, say, builds.hex.pm.

3. A builds/{os}-{arch}.txt file in the git repository. It would be automatically updated by build job. The format could be similar to https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/builds.txt. Tools like asdf and erlef/setup-beam could easily fetch this file, https://github.com/{user}/{repo}/raw/{branch}/{file}, to know all available builds.

The builds would ideally be triggered by github.com/erlang/otp which is very easy to do.

Hosting this under https://github.com/erlef is not strictly necessary, all compute and storage can be done on GitHub free plan. Given the non-profit status, it mighe be easier to ask for more compute and storage, if needed.

Alternative Solutions

• Focus on beammachine.cloud.

• Focus on kerl, perhaps same idea like above but add https://github.com/kerl/otp_builds.

• Add GitHub Actions build job to https://github.com/hexpm/bob and publish builds to builds.hex.pm. We (Hex team) are OK with that and it would match existing Ubuntu builds we maintain. As mentioned previously though, we'd prefer to reduce instead of increase infrastructure that we maintain. builds.hex.pm is powered by an Elixir app, S3 bucket, and (a fair amount of) Fastly VCL scripts.

Possible Future Work

• Add Ubuntu builds. Deprecate Hex-maintained Ubuntu builds in favour of these.

• Add builds for other Linux distributions. Ideally its builds are not distribution specific but libc specific, ie glibc vs musl.

Proof of Concept

https://github.com/wojtekmach/otp_builds

• releases: https://github.com/wojtekmach/otp_builds/releases
• builds/*.txt: https://github.com/wojtekmach/otp_builds/tree/main/builds
• build script: https://github.com/wojtekmach/otp_builds/blob/main/scripts/build_otp_macos.sh
• build job: https://github.com/wojtekmach/otp_builds/blob/main/.github/workflows/build.yml
• update builds/*.txt job: https://github.com/wojtekmach/otp_builds/blob/main/.github/workflows/update_builds.yml

[wojtekmach]

I previously submitted a PR to OTP: erlang/otp#5723. This is another attempt at a community maintained solution.

[josevalim]

👍 this type of project makes perfect sense to be hosted here, since it benefits a big part of the community (and if we were to host Ubuntu/Debian builds, that’s a huge majority).

[paulo-ferraz-oliveira]

@wojtekmach, great initiative. Me and @starbelly started https://github.com/jelly-beam/otp-macos a while ago, with the goal of eventually (after testing and preliminary experimental integration into setup-beam) moving it to erlef. It runs on GitHub, where the images are released, and makes considerations around OpenSSL. We also include a builds.txt (setup-beam compatible) and would generate images every x hours. We have the goal (check the issues) to replace as much 3rd party as possible, to make it as independent of dependencies as possible. Would some of it fit this issue's goals? Or could we even discuss moving this to erlef and build on top of it?

[rvirding]

A very brief comment is that if possible maybe support older OTP versions. I sometimes need to run different versions. I don't know how much effort this would be.

[ferd]

Something interesting to at least specify here is that since it'll use static linking, you should specify what your policy will be with regards to upgrading openssl versions in images after security advisories. The reason for this is that if you keep old images static, you're probably fine for CI purposes, but it's going to be a risk for releases installed on user systems because vulnerable libraries may be linked into the app.

So either the builds would need to be re-built, or a clear warning explaining this limitation should be mentioned.

[starbelly]

Something interesting to at least specify here is that since it'll use static linking, you should specify what your policy will be with regards to upgrading openssl versions in images after security advisories.

Indeed and this was something I wanted to figure out before anything was finalized. In the worst scenario every build has to be invalidated, and a massive re-build must happen, which means service interruption if you will. I think we need to ask the question again : Why do we want to statically link? The main reason I thought about the time was so you that your build was portable, but that's not exactly going to be true is it? Even if you statically link, you're still going to have other dependencies (libc and friends). Playing devil's advocate to myself here, but yeah.

[paulo-ferraz-oliveira]

In the worst scenario every build has to be invalidated

You can probably build a new release, with a newer OpenSSL, and replace the reference in the list of builds. Consumers just need to know they shouldn't cache endpoints to releases. e.g. if Hex/Bob replaces a build in builds.txt, today, used by setup-beam (results may vary for asdf) service is not interrupted.

And are any of these considerations done, at present, for the builds setup-beam uses already, for Ubuntu (Hex/Bob) and Windows (erlang/otp)?

[wojtekmach]

My understanding is macOS ships with LibreSSL but without header files and there is a recommendation to bring your own SSL library anyway: https://rentzsch.tumblr.com/post/33696323211/wherein-i-write-apples-technote-about-openssl-on.

In my testing the build is quite portable. I didn't do a thorough testing but I tried it on a few laptops and fresh VMs and it worked. My understanding is there's no glibc/musl on macOS, there's one libc (libSystem) that Apple maintains.

One of the goals I listed is:

Statically linked OpenSSL and wxWidgets.

And to be fair that's not quite what I am after, or rather, that's not a goal but a solution to reach a goal. I want to download an OTP on my Mac that just works, I don't have to install anything else (no homebrew, macports, etc). I'd personally be OK with OTP wx app not working out of the box (e.g. you need to install hombre and brew install wxwidgets) but I think it'd unnecessary friction. A lot of learning resources mention observer.

Does anyone know what is OTP security policy for Windows builds?

[wojtekmach]

@rvirding how far would you prefer to go back? I would consider OTP 25 as the bare minimum. Pertinent to builds, it adds JIT for arm64, OpenSSL 3.0 support, and relocatable installation directory.

[wojtekmach]

I have updated the proposal with a proof of concept implementation, currently under my personal account:

https://github.com/wojtekmach/otp_builds

• releases: https://github.com/wojtekmach/otp_builds/releases
• builds/*.txt: https://github.com/wojtekmach/otp_builds/tree/main/builds
• build script: https://github.com/wojtekmach/otp_builds/blob/main/scripts/build_otp_macos.sh
• build job: https://github.com/wojtekmach/otp_builds/blob/main/.github/workflows/build.yml
• update builds/*.txt job: https://github.com/wojtekmach/otp_builds/blob/main/.github/workflows/update_builds.yml

[paulo-ferraz-oliveira]

Does anyone know what is OTP security policy for Windows builds?

I have no specific knowledge about this, but erlang/otp is a good place to ask (or maybe the Erlang Forum?) and information would be more acessible for future reference.

[garazdawi]

Does anyone know what is OTP security policy for Windows builds?

We don't update installers after they have been released. Since we only depend on libcrypto its very rare for there to be vulnerabilities that effect Erlang.

[cocoa-xu]

@rvirding how far would you prefer to go back? I would consider OTP 25 as the bare minimum. Pertinent to builds, it adds JIT for arm64, OpenSSL 3.0 support, and relocatable installation directory.

In my otp-build repo, it can go back to OTP 24.0 for x86_64 macOS and 24.3 for arm64 macOS. I also have support for some BSD systems.

[wojtekmach]

@cocoa-xu I didn’t know you maintain OTP binary builds too but can’t say I’m surprised. This is awesome!

I noticed you have Linux builds per libc, not per distribution, which is something @tsloughter was keen on.

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead? How do you approach OpenSSL and wxWidgets?

[wojtekmach]

I have added beamup to the use case list in this proposal, congrats on the release @tsloughter! :)

[cocoa-xu]

I noticed you have Linux builds per libc, not per distribution, which is something @tsloughter was keen on.

Yeah, and part of the reason for this is for containers, especially for people who want to use alpine ones which are using musl libc.

How do you approach OpenSSL and wxWidgets?

For OpenSSL I also have a repo that's monitoring the latest OpenSSL version and does the precompilation workflow here, https://github.com/cocoa-xu/openssl-build (although iirc on macOS it might not be able to do a full statically-linked build); As for wxWidgets, it's not included yet (as I was mainly targeting for CI) but I believe it won't be too hard to statically link that.

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead?

I think our goals definitely would align, and I'll be happy to add support for wxWidgets and generate a Hex/Bob compatible builds.txt file!

[tsloughter]

@wojtekmach thanks! still needs to have support for Erlang binary release install but that is up their on my priority list after finishing up the missing commands to be feature equivalent to erlup.

@cocoa-xu ah, your per-libc for alpine makes sense.

I was thinking about just setting up builds per-distro and contributing that since it wouldn't be too bad with docker. Starting with Fedora since that is what I use.

I doubt I'll be able to catch for a user when an install breaks because they upgraded distro versions though. Usually they won't, even though library versions installed changed, it'll keep working. But when there is a change that breaks the build it would be so nice to not just crash but be able to say, "maybe you should reinstall since OTP-27.0 was built for Fedora 39 not your current 42".

But I'm execve'ing the executable of commands like erl, not going to be able to recover a crash there.

Maybe just a command of upgrade-possibly-outdated-installs so I don't have to actually keep track of what versions of what cause breakage and leave it up to the user to run if they think it might help, hehe.

[wojtekmach]

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead?

I think our goals definitely would align

@cocoa-xu after thinking about it some more, I think the end goal is something like your project where we support as many targets as possible, but I think it's better to start small after all. As was mentioned here previously, with macOS support we will serve a good chunk of the community and down the road with Ubuntu probably a majority. I'm committing to maintaining macOS builds as I am a user. I'd commit to Ubuntu too (since it's moving responsibilities away from Hex/Bob) but that's about the extent I'm comfortable with at the moment.

[wojtekmach]

I've just opened the PR with the implementation of this proposal: erlef/otp_builds#1

[paulo-ferraz-oliveira]

@wojtekmach, would it make sense to refer kerl (https://github.com/kerl/kerl) in the use cases? It's what asdf uses under the hood, but I'm not sure if the end goal would be to bypass that.

Regarding setup-beam, at least rebar3 would greatly benefit testing with it. At the moment, it's done with brew, as a best effort but it's broken builds before, e.g. here and here.

In "Existing Solutions" it'd probably make sense to explicitly mention Nix (?) https://github.com/NixOS/nixpkgs/blob/nixos-24.05/pkgs/development/interpreters/erlang/generic-builder.nix#L189, especially if sharing compilation flags across these multiple tools would increase consistency for the ecosystem.

[wojtekmach]

Hey everyone, I believe https://github.com/erlef/otp_builds is ready! Here are some changes from the original prototype:

• build tarballs are available under URLs like: https://github.com/erlef/otp_builds/releases/download/{ref}/otp-{target}.tar.gz. {ref} is e.g. OTP-27.1.2 and target is aarch64-apple-darwin or x86_64-apple-darwin. Example URL: https://github.com/erlef/otp_builds/releases/download/OTP-27.1.2/otp-aarch64-apple-darwin.tar.gz

• because build tarball no longer has OTP version in the filename, we can trivially fetch the latest build (corresponds to latest stable release): https://github.com/erlef/otp_builds/releases/latest/download/otp-aarch64-apple-darwin.tar.gz

• OTP 25+ is supported. See https://github.com/erlef/otp_builds/tags for a list of all supported releases.

• master-latest, maint-latest, maint-25-latest, maint-26-latest, maint-27-latest releases are available that are rebuilt daily and they track the corresponding master, maint, etc branches from erlang/otp.

• "builds.txt" is now "builds.csv": https://github.com/erlef/otp_builds/blob/main/builds/aarch64-apple-darwin.csv

• we're now using GitHub artifact attestation:

  $ gh attestation verify --repo erlef/otp_builds \
      <(curl -fsSL https://github.com/erlef/otp_builds/releases/latest/download/otp-aarch64-apple-darwin.tar.gz)
  Loaded digest sha256:14d65f1f918cc91d42318d72dcd07f8c8439d25385fba2ac5b7a87bc51d6ccb5 for file:///dev/fd/11
  Loaded 1 attestation from GitHub API
  ✓ Verification succeeded!
  
  sha256:14d65f1f918cc91d42318d72dcd07f8c8439d25385fba2ac5b7a87bc51d6ccb5 was attested by:
  REPO              PREDICATE_TYPE                  WORKFLOW
  erlef/otp_builds  https://slsa.dev/provenance/v1  .github/workflows/build.yaml@refs/heads/main
  

• we now have a scheduled job that fires every 15min and checks whether there was a new OTP release and if so, it schedules a build.

There are still some outstanding issues and PRs in https://github.com/erlef/otp_builds but I believe the project can be considered a v1. I think we can close this and continue discussions on individual issues and PRs in erlef/otp_builds!