GitHub-hosted arm64 runners

[Qard]

Many devs have migrated to M1 laptops and there's a growing movement of migrating servers to arm64 linux. It would be great if there were official arm64 runners. Custom runners is not a great solution for something that is rapidly becoming the default case.

[rochdev]

It's worth noting that most competitors like for example CircleCI already support this. Heck, even Travis supports this.

[a-h]

The Github Actions home page makes it look like ARM is supported as a 1st class feature, when it clearly isn't.

https://github.com/features/actions

I don't even see ARM or ARM64 on the roadmap at https://github.com/orgs/github/projects/4247/views/1?filterQuery=ARM

[jonesmz]

This also confused and annoyed me. Its the whole reason I'm even on this post because I was trying to figure out why I was lied to by the section heading on their advertising copy.

[mustafaergul]

Travis CI has supported the Arm64 for some time, I think worth to try:
https://docs.travis-ci.com/user/multi-cpu-architectures

[lukasz-mitka]

You can try this https://github.com/docker/setup-qemu-action in the meantime.
Won't be very efficient and may not work for everything, but you should be able to build arm docker image for example.

[edhemphill]

Could you not just run the aarch64 github runner in a VM Linux running on M-series Mac?

[lukasz-mitka]

The pricing for Mac runners is ridiculous
https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates

Linux 2 $0.008
macOS 6 (M1) $0.16

Even after adjusting for number of cores it's over 6 times more expensive.
So I guess if qemu is slower than that it could work. Did anyone try it?

[JanneKiiskila]

You really do not need an Apple/Mac for Arm64/ aarch64 jobs. You could use some arm64-server instead, which would be very, very cost-efficient (due to way smaller power consumption than Intel/AMD alternatives).

• AWS has the Gravitron instances, they are very cost-effective.
• Those are of course AWS specific, but you can get for example the Ampere solutions.
• Azure (by Microsoft) already has those and since GitHub belongs to Microsoft nowadays as well, they should be able to leverage those? Ref: https://azure.microsoft.com/en-us/blog/azure-virtual-machines-with-ampere-altra-arm-based-processors-generally-available/

Would be very neat to get some update from someone in GitHub and/or Microsoft to comment on what's the hold-up here?

[Qard]

You do need a mac target if you're trying to test for M* Apple devices. Given their ubiquity with developers there should definitely be runners for that.

[lukasz-mitka]

It's now GA for 'Team and Enterprise Cloud plans'
https://github.blog/changelog/2024-09-03-github-actions-arm64-linux-and-windows-runners-are-now-generally-available/

[maybedino]

Almost every cloud provider, including Azure, has arm64 servers now. But building arm64 container images on Github still requires QEMU which is so slow, it is close to being unusable. A ~2 minute amd64 build can take 30+ minutes on arm64. Native arm64 runners would be great, as more people are switching to arm64 servers and Github is lagging behind here.

[lukasz-mitka]

Well, arm64 doesn't have AI buzz to it so GitHub doesn't care :(

[JanneKiiskila]

Well, Arm64 should be perfectly fine for that. You can run amazing stuff even on the Arm microcontrollers.

• tinyML etc.
• https://alifsemi.com/ does also the Ensemble chips with AI accelerators.

[edhemphill]

All next generation AI purpose built chips will use aarch64 CPUs as part of the entire unit. i.e. you wont need an x86 CPU in there. Example: https://www.anandtech.com/show/20001/nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory

GH200 uses an Arm Neoverse v2 CPU core. Google, Amazon doing the same stuff. All public info.

[janeklb]

Is it possible to update this discussion's title to be "Github-hosted arm64 runners"? I think that's what most people are asking for (since it's been possible to run self-hosted arm64 runners for a good while now). cc @Qard

[lukasz-mitka]

Arm-based hosted runners are coming to GitHub Actions!
[...]
These runners will be entering private beta in January 2024.

https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

Finally!

[debu99]

just private beta? i guess still need wait until middle of 2024 to use

[mio-19]

I guess we need to wait until 2025

[jbrsb]

Hi, does anyone have a date or any news on this feature ?

[relistan]

Really want this ASAP

[matti]

just private beta? i guess still need wait until middle of 2024 to use

@debu99 middle of 2024 might be too optimistic

[Jolg42]

I think this can be closed now? 🤔

EDIT: For macOS only:

See https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

Today, GitHub is excited to announce the launch of a new M1 macOS runner! This runner is available for all plans, free in public repositories, and eligible to consume included free plan minutes in private repositories. The new runner executes Actions workflows with a 3 vCPU, 7 GB RAM, and 14 GB of storage VM, which provides the latest Mac hardware Actions has to offer. The new runner operates exclusively on macOS 14 and to use it, simply update the runs-on key in your YAML workflow file to macos-14.

See https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available/

The macOS 14 runner image is now available for GitHub hosted runners. Workflows executed on this image will run exclusively on the 3 vCPU M1 runner announced earlier today. To use the runner, simply update the runs-on: key in your YAML workflow file to macos-14, macos-14-xlarge, or macos-14-large.

I tried it and it works very well ✨
Thanks to the people at GitHub, this was probably a monumental effort behind the scenes 🙌🏼

[Feichtmeier]

this is about linux arm runners

[Jolg42]

Right, it was confusing, because it was mentioning M1 😄

[WeiChungHsu]

Can I use macos-14 runner with the service container such as debian or ubuntu?
I want to build arm64 linux kernel and/or arm64 architecture specific codes, but NOT to limited in macos.

I search github document and it says the service container only works with ubuntu runner. Does it mean macos-14 arm64 runner can't work with service container?

[bpcreech]

I wonder if you could brew install colima on the macos runner, and run an aarch64 Ubuntu container on that.

Not easy and probably not very efficient, but probably more efficient than using qemu to emulate aarch64 on x86_64...

[Rhys-T]

I wonder if you could brew install colima on the macos runner, and run an aarch64 Ubuntu container on that.

Apparently not - the Apple Silicon runners don't support nested virtualization because of limitations in Apple's frameworks.

[nebuk89]

Hey! Yes ARM is currently in private beta while we work on ensuring we have an image that provides a good set of tools and we get our capacity all sorted :) it will be nearer the end of this year that we get our new default labels supporting ARM - but we are working on it actively <3

[nebuk89]

Good question, at the Org level we are aiming for June/July. For individual accounts later in the year (hopefully no later than end of Nov but depends on capacity and how many people use them!)

[jacobsvante]

June/July sounds promising!

[LouisBoudreau]

Thanks @nebuk89 for your reply!

[qwertychouskie]

@nebuk89 Any chance the Caruso-dev org can get added to the beta as well? arm64 is currently a blocker for us moving our deployment script for one of our repositories to Github Actions rather than running on a developer's system.

[zentavr]

July is here :)

[relistan]

I'm personally looking for linux ARM64 runners, so macOS runners don't do much for me at the moment.

We're using ARM64 on everyone's desktop and ARM64 in production and GitHub can't build images for that chain without emulating ARM64 on AMD64.

[crohr]

The RunsOn tool supports both arm64 and x64 and is a drop-in replacement. Might be worth a look.

[robwafle]

How can someone get included in the private beta? I could use an ARM64 Ubuntu runner today.

[lukasz-mitka]

There's a link at the end of this post
https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

[filipRisteski]

I went with ARM deployment for production, and I am very surprised that Github doesn't support this.

[kei1987]

Arm64 for macos runner is available but docker is not installed, and nested virtualization is not possible. No way to build a docker image in arm64 architecture as of now?

[patrick-stephens]

This is clarified in the statement that comes along with using the beta explaining that it is not included. You can install docker like any other ubuntu server though - we did that to verify the builds work fine. My understanding is that public beta will likely ensure this is included.

[kei1987]

yep, we are actually using ubuntu-latest with QEMU instead. It can build an image for arm64 arch but takes over 15 minutes to complete, which is not ideal for CI. Let's wait for the public beta

[patrick-stephens]

Ah sorry, you are talking about macOS runners not the Ubuntu ARM ones.
We're just building ARM images on the Ubuntu ones, not macOS. Containers on mac always needs more effort so never bother :)

[SanderVocke]

While we wait for publicly available Linux arm runners, I have made a marketplace Action which can run entire workflow runs transparently in a QEMU-emulated container on top of a x86_64 runner, for a "close-to-built-in" experience that doesn't require VMs. It has limitations for sure, but could be a solution for some projects:

https://github.com/SanderVocke/setup-qemu-container

[maybedino]

For docker, if your language supports cross-compilation (like go), or produces platform-independent code (like c# or ts), you can use multi stage builds to build on the runner architecture and just copy the result into the final image like this:
https://docs.docker.com/build/building/multi-platform/#cross-compilation
Which will be much faster.

More details: https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

[alec-pinson]

For docker, if your language supports cross-compilation (like go), or produces platform-independent code (like c# or ts), you can use multi stage builds to build on the runner architecture and just copy the result into the final image like this: https://docs.docker.com/build/building/multi-platform/#cross-compilation Which will be much faster.

More details: https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

Thanks @maybedino this gave a massive improvement for me!

before

#32 [linux/arm64 build_base 10/10] RUN cd cmd/frontend &&     go build -o /build/out/my-app .
#32 DONE 292.0s

after

#27 [linux/amd64->arm64 build_base 10/10] RUN cd cmd/frontend &&     GOOS=linux GOARCH=arm64 go build -o /build/out/my-app .
#27 DONE 41.8s

[SanderVocke]

We've just been using https://github.com/docker/setup-qemu-action for quite a while now but the issue is whilst this works it is super slow on emulated targets.

In many cases that may be the more straightforward choice. In my case I want to build a complicated Python package in a manylinux container, where the entire container should be ARM. In particular, I want to run multiple GHA workflow steps in the same emulated container, which I think is not possible with setup-qemu-action without explicitly using qemu in every step.

I don't want to build my software as a Docker image, as that would involve migrating a complex set of Github Actions scripts to a Dockerfile.

[crohr]

@SanderVocke is your repo open-source? In which case you can setup RunsOn for free and benefit from the fastest ARM processors around - https://runs-on.com/reference/benchmarks-gha-providers/#arm64-runners

[SanderVocke]

@SanderVocke is your repo open-source? In which case you can setup RunsOn for free and benefit from the fastest ARM processors around - https://runs-on.com/reference/benchmarks-gha-providers/#arm64-runners

Wow, thanks, I'll check it out! Honestly, I had been looking for something like this before making the emulation wrapper. But I couldn't find any runner hosts that offered free plans for open-source projects. Great to know that it is out there and I will definitely give it a try.

[theofficialgman]

@ankneis I have noted the roadmap for github hosted ARM64 runners currently say they will be "available for GitHub Team & Enterprise plans only." . What are the dates for full availability? Or is this a typo and ARM64 Linux runners will be available to all github plans like MacOS ARM64 runners?
github/roadmap#955
github/roadmap#960

[LewisCowlesMotive]

The weird thing about that is that they should be significantly cheaper to operate... Maybe in Startup mode they had more ability to go out and Raise than being owned by a multi-billion dollar company.

[air3ijai]

Arm-based hosted runners are coming to GitHub Actions! 🚢 #73040

[kjxbyz]

https://github.blog/changelog/2024-06-03-actions-arm-based-linux-and-windows-runners-are-now-in-public-beta

[jacob-carlborg]

Does anyone know if they support hardware accelerated nested virtualization (KVM)?

[LewisCowlesMotive]

@jacob-carlborg experimenting with Firecracker, or similar neo-VM tech?

[jacob-carlborg]

@LewisCowlesMotive close. I'm the author of this GitHub action: https://github.com/marketplace/actions/cross-platform-action. It uses virtualization and emulation to allow running jobs/commands on platforms not natively supported by GitHub. The ARM64 platforms are currently using emulation. Would be faster with hardware accelerated virtualization.

[crohr]

@jacob-carlborg Nice! I'd love to improve the benchmark of GitHub Actions providers that I maintain, with indications on support of hardware accelerated virtualization for the various providers (including GitHub). Does your action somehow expose an output that could be used to quickly know if the runner supports it for one of the architectures supported by your action?

[jacob-carlborg]

@crohr no, not in that way. Now since the Ubuntu x86-64 runners do support hardware accelerated virtualization it just assumes that and tries to use KVM. I guess you can use the action and it will get an error if KVM is not available. It's using a custom built QEMU, only available for x86-64 for the host platform. So it won't work on ARM64 runners.

[TaxBusby]

Is anyone else experiencing longer-than-normal times waiting for runners to come online using the arm64 runners? We're using the lowest spec'd 2-core runners, and routinely I see jobs where there are multiple minutes before a job begins.

Here is a job that took 44 seconds to run, but the "total time" was almost 5 minutes, which was all waiting for the runner to become available.

Edit: This particular case might have been during a github incident, which caused a longer than normal delay. But I still feel like my arm64 runners take longer than x86 to become available and retrieve jobs.

[jacob-carlborg]

If you misspell the name of the runner it will never start and it won't tell you it can't find a matching runner. Double check that the name is correct.

[patrickelectric]

If you misspell the name of the runner it will never start and it won't tell you it can't find a matching runner. Double check that the name is correct.

If you check the yml file in my previous comment and the configuration screenshot, you'll see that the name is correct: "ubuntu-24-arm", if not, can you point where ?

[jacob-carlborg]

If you check the yml file in my previous comment and the configuration screenshot, you'll see that the name is correct: "ubuntu-24-arm", if not, can you point where ?

Looks correct to me. I just mentioned it as one possible problem. I think it's very poor UI from GitHub and I've run into this myself, having misspelled a runner.

[patrickelectric]

In the end was a configuration of the runner where by default it does not have allowed repositories, it's necessary to select the repositories manually or allow all.

[patrick-stephens]

Ah yeah, we have the same to only allow it for certain repos.

[kjxbyz]

GitHub Actions: arm64 Linux and Windows runners are now generally available

https://github.blog/changelog/2024-09-03-github-actions-arm64-linux-and-windows-runners-are-now-generally-available/

[dmtrKovalenko]

*for Team and Enterprise accounts

[kjxbyz]

However, it is also an important sign

[jontze]

Good to read this! :)

Does anyone know if there are plans to make the ARM runner available for personal accounts (perhaps only for paying users)?

I have an open-source project that could really benefit from this feature.

[jacob-carlborg]

Does anyone know if there are plans to make the ARM runner available for personal accounts (perhaps only for paying users)?

I have an open-source project that could really benefit from this feature.

From the blog post: "We expect to begin offering Arm runners for open source projects by the end of the year" https://github.blog/news-insights/product-news/arm64-on-github-actions-powering-faster-more-efficient-build-systems/#get-started-using-arm-hosted-runners-today.

[lkfortuna]

Hi there from Actions! - to answer the question, we just released the current arm runners into GA and we do plan to release arm runners for open source at EOY. Stay tuned, it is on the roadmap.

[ssbarnea]

Added actions/partner-runner-images#22 as the label/runner names seems not to be documented.

[air3ijai]

At this stage stage, we are adding labels by ourselves when add a runner, on the screenshot above.

And free runners probably will have their labels, like you've mentioned - ubuntu-22.04-arm64?

[lkfortuna]

To clarify - at this time the arm64 runners available are not free in public, they are larger runners and so must be created via runner groups. This is all specified in the original blog post with the announcement.

The upcoming release will add arm64 as another option for our standard hosted runners - which means they will be free in public repositories and there will be a label to target them.

[pl4nty]

arm runners for open source at EOY

Arm64 GitHub-hosted runners on GitHub Free plans ... will be released broadly in early 2025

https://github.blog/news-insights/product-news/universe-2024-previews-releases/

😢

[pbo-linaro]

This will be awesome to have access to arm64 for public repo (even though it seems it has been delayed to next year - early 2025 - from link in previous message).
@lkfortuna Will windows-arm64 images will be available too?

[skyblaster]

First, thanks in advance for these new runners. I can't wait to try them!

As someone who struggles to implement a manifest for container packages pushed to ghcr.io, and also not end up with an "unknown/unknown" arch in the process, I really hope new OCI spec parallel-build templates can be added to https://github.com/<user>/<project>/actions/new

Bonus points for additional examples using redhat-actions/buildah-build and redhat-actions/push-to-registry

[johnnynunez]

When it will be free?

[air3ijai]

https://github.blog/changelog/2024-06-03-actions-arm-based-linux-and-windows-runners-are-now-in-public-beta

Arm runners are available to customers on our Team and Enterprise Cloud plans. We expect to begin offering Arm runners for open source and personal accounts by the end of the year.

[johnnynunez]

https://github.blog/changelog/2024-06-03-actions-arm-based-linux-and-windows-runners-are-now-in-public-beta

I was hoping for an earlier date, because software like OBS-Studio want to add it to create their windows arm releases as well.

[LewisCowlesMotive]

@lkfortuna or other GitHub folks. Are the arm64 runners documented, in the same way that the windows runners are?

I'm interested to know if there is deeper technical information about the choices made for each hardware platform or architecture. I'd be surprised if packer for example can be used (it is for Windows x86_64), because virtualbox doesn't yet support arm64. But maybe packer works with other backends? I'm so interested in learning more about this.

[jacob-carlborg]

Yes, Packer works with several other backends. One of the backends it supports is QEMU, which runs on most platforms and can emulate/virtualize most platforms. What is your goal, build virtual machine images for Windows ARM64? QEMU can run on x86-64 and build an ARM64 image. It will be emulated so it will be slower compared to virtualization. It's not clear yet if the ARM64 runners support hardware accelerated nested virtualization. The macOS ARM64 runners don't. If they don't, you need to use emulation and then it doesn't really matter on which platform you run QEMU, it will most likely be equally slow (assuming the same hardware) regardless.

I have several projects which builds ARM64 images using Packer with QEMU as the backend. Here's one example [1]. That file is actually both for ARM64 and x86-64. It's built with GitHub Action using the regular x86-64 Linux runners.

[1] https://github.com/cross-platform-actions/freebsd-builder/blob/master/freebsd.pkr.hcl