.NET AWS Lambda function using .NET 7 RC1 fails to start when using a custom runtime on AWS Linux 2 on arm64

[martincostello]

Description

This issue is a duplicate of aws/aws-lambda-dotnet#1310 that I'm also opening for visibility in case some collaboration is needed between Microsoft and AWS to resolve this issue to enable .NET 7 on AWS Lambda.

I attempted to deploy a .NET Lambda function I have updated from .NET 6 to .NET 7 to AWS Lambda using a custom runtime (deployment).

However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in AWS CloudWatch show that the function is failing to start.

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.

I'm not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I'm all ears to unblock early adoption.

/cc @normj

Reproduction Steps

An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.

Expected behavior

The .NET Lambda function starts successfully.

Actual behavior

The Lambda function fails to start with the following error:

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

Regression?

To the end-user, yes, but the change in the GLIBC version is an intentional change.

Known Workarounds

None.

Configuration

• .NET SDK: 7.0.100-rc.1.22431.12
• Operating system: Amazon Linux 2 on Graviton/arm64 (Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022)

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

.NET on arm64 requires a Linux distro with glibc 2.27 or newer. It roughly matches to Ubuntu 18.04 or newer.

This is documented at https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md

[jkotas]

https://repost.aws/questions/QUrXOioL46RcCnFGyELJWKLw/glibc-2-27-on-amazon-linux-2

[normj]

@jkotas Can you confirm that 2.27 is only required for ARM and not X64? Out of curiosity what was the feature in 2.27 that was needed for .NET 7 as this is potentially going to block a fair amount of use cases on AWS.

[martincostello]

@normj Some digging was done by @slang25 earlier when we found this issue with Lambda, and this appears to be the main motivation: 31cd7eb

[jkotas]

The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.

CentOS 7 that we use to build for x64 does not work well on Arm64. And Ubuntu 16 that we use to build for arm64 in .NET 6 is out of mainstream support.

The change you have linked was just a follow up cleanup.

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

This issue is a duplicate of aws/aws-lambda-dotnet#1310 that I'm also opening for visibility in case some collaboration is needed between Microsoft and AWS to resolve this issue to enable .NET 7 on AWS Lambda.

I attempted to deploy a .NET Lambda function I have updated from .NET 6 to .NET 7 to AWS Lambda using a custom runtime (deployment).

However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in AWS CloudWatch show that the function is failing to start.

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.

I'm not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I'm all ears to unblock early adoption.

/cc @normj

Reproduction Steps

An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.

Expected behavior

The .NET Lambda function starts successfully.

Actual behavior

The Lambda function fails to start with the following error:

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

Regression?

To the end-user, yes, but the change in the GLIBC version is an intentional change.

Known Workarounds

None.

Configuration

• .NET SDK: 7.0.100-rc.1.22431.12
• Operating system: Amazon Linux 2 on Graviton/arm64 (Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022)

Other information

No response

Author:	martincostello
Assignees:	-
Labels:	question, area-Meta
Milestone:	-

[jeffschwMSFT]

closing this issue as answered, but we can reopen if necessary.

[sebpop]

@jkotas wrote:

The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.
CentOS 7 that we use to build for x64 does not work well on Arm64. And Ubuntu 16 that we use to build for arm64 in .NET 6 is out of mainstream support.

Amazon Linux 2 is based on CentOS with a glibc older than Ubuntu 18.04:

[al2] $ /lib64/libc.so.6 
GNU C Library (GNU libc) stable release version 2.26, by Roland McGrath et al.

[ubuntu 18.04] $ /lib/x86_64-linux-gnu/libc.so.6 
GNU C Library (Ubuntu GLIBC 2.27-3ubuntu1.6) stable release version 2.27.

Could the .net framework be built for arm64 on AL2 that has the oldest version of glibc?
AL2 is supported on ARM64 and will be supported until at least 2024-06-30.

From https://aws.amazon.com/amazon-linux-2/faqs/

Amazon Linux 2 end of support date (End of Life, or EOL) has been extended by one year from 2023-06-30 to 2024-06-30

Thank you,
Sebastian

[AdamTylerLynch]

From the supported OS versions page, I note that you compile on 2.17 for x64 (CentOS7), but 2.27 on Ubuntu.

If you could compile on 2.26 that will fix the issue.

• x64: glibc 2.17 (from CentOS 7)
• Arm32, Arm64: glibc 2.27 (from Ubuntu 18.04)
• Alpine (x64 and Arm64): musl 1.2.2 (from Alpine 3.15)

[normj]

+1 @AdamTylerLynch push for using 2.26. AL2 is heavily used across AWS services particular across the managed compute services. There are plans to move to newer versions of Linux but it will take a quite awhile for that to replace the usage of AL2.

[richlander]

There are two aspects at play here:

• .NET on Arm64 requires Linux kernel 4.14+ to work correctly and is only supported in that configuration.
• We switched to running on Ubuntu 18.04 for compiling the product (hence the glibc issue).

For fairness, .NET likely works most of the time on <4.14 Arm64 machines, but there are some scenario where it won't. It relates to memory barrier usage. @jkotas likely has more insight. The current product only enables .NET to run in places where it will work correctly.

Context: https://devblogs.microsoft.com/dotnet/announcing-net-core-3-0/#platform-support

[slang25]

Thanks @richlander. So if I understand correctly, there's no version of Ubuntu lower than 18.04 that ships with a version of the kernel 4.14+
https://ubuntu.com/kernel/lifecycle

and Ubuntu 18.04 ships glibc 2.27

[normj]

AL2 is currently running 4.14 by default, at least in Lambda but is using GLIBC 2.26. So we are unfortunately in an awkward spot.

[richlander]

@slang25 -- That's my understanding.

[jkotas]

Linux kernel 4.14+ is another minimum requirement. It is separate from the minimum glibc version requirement.

I am sorry we are not able to downgrade the Linux Arm64 build environment for Microsoft-built .NET 7 runtime to lower the minimum required glibc version. The alternatives and workarounds include:

• Stay on .NET 6
• Run on x64
• Build your own runtime from source

[normj]

Thanks @jkotas, I knew it was a long shot especially at this late point in the .NET 7 process but still had to ask.

[jkotas]

The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.

I should have been more precise here. We want the distro that we use as a build environment to have enough support life left. It means one year or more. It does not make sense to ship using build environment that is nearing end of life, and then switch to a different build environment after several months. It would just create unnecessary instability in servicing.

[richlander]

We’ve been discussing this more and want to start on a solution to this problem for .NET 8. We can investigate if there is an option to backport to .NET 7, however we'll have to satisfy a bunch of risk concerns for that so no promises there.

The basic problem is that we want to use modern distros for our build environment, while also enabling breadth targeting. The libc version ends up being the primary challenge in doing that, as has been described in this issue. We’ve already been thinking about that, but didn’t fund the solution for .NET 7. We also ran into this same problem with #70111 (comment). We were planning to tackle it for .NET 8 but now have extra motivation because of this scenario. We believe that rootfs targeting is the best direction, which we already use in some scenarios. It would be great to get your viewpoint on that.

For .NET 7, we’re seeing this challenge with Arm64. For .NET 8, we won’t be able to rely on our CentOS 7-based x64 build any longer (EOL June 2024) and so will have this same problem with X64.

We’ve previously discussed minimum libc targeting with Canonical and Red Hat. Our goal is to target the lowest version that is needed. That kind of coordination requires significant ahead-of-time planning and we’re happy to include Amazon in that. I'll start a new issue on this topic assuming folks are interested in working on it.

[normj]

Thanks @richlander for continuing the discussions. I would appreciate being included in the coordination as Amazon Linux 2 is going to kick around for awhile. I am very concerned about the future path with .NET 8 especially if X64 is also a concern and I want to get ahead of the upcoming issues as soon as possible.

[richlander]

We will use #69361 for further discussion on this topic.