Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preliminary work to compile on Amazon Linux 2023 #122

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Conversation

mnapoli
Copy link
Member

@mnapoli mnapoli commented Sep 18, 2023

AL2023 is eventually going to happen. Out of curiosity, I drafted this PR trying to build our layers for AL2023.

It removes a lot of custom system libraries we compile because AL2023 contains more recent versions: https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.1.html

We'll need to reevaluate when we actually get AL2023 Lambda images (right now we have the generic AL2023, not specific to Lambda), but it is looking good.

I'm opening this PR to gather early feedback and see if I'm missing something.

# - libzip: minimum required CMAKE version 3.0.
RUN LD_LIBRARY_PATH= yum install -y cmake3
# Override the default `cmake`
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmake3 is installed by default

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is due to AL2 having both cmake and cmake3 packages to ensure that software needing cmake that didn't yet work with cmake3 could continue to be built.

For AL2023, it's all cmake version 3.

# - curl
# - php
RUN yum install -y perl-IPC-Cmd
ENV VERSION_OPENSSL=3.0.10
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 3.0.8

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security issues in the native version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried asking around but have no answers yet. Do you know if the versions compiled in AL might have some security fixes applied?

I'm just surprised AL would ship with security issues, how do all customers do? I wouldn't expect everyone to compile everything from source, right?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We backport security fixes to the OpenSSL versions that we ship. See https://alas.aws.amazon.com/alas2023.html for updates.

For example: https://alas.aws.amazon.com/AL2023/ALAS-2023-306.html and https://alas.aws.amazon.com/AL2023/ALAS-2023-222.html are some recent ones.

If there's anything from newer OpenSSL that you need, please do let us know and submit an issue to https://github.com/amazonlinux/amazon-linux-2023/ for us to look at.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a version that supports quic possibly, for http/3 in curl, also. I am not sure how long it takes AWS to apply security fixes. I like the flexibility that keeping this gives us to easily re-compile, or for me to fork and re-compile to get something patched immediately.

# Needed by:
# - php
# - libnghttp2
ENV VERSION_XML2=2.11.5
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 2.10.4

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security issues in the native version?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do backport security fixes to the version we ship in Amazon Linux. See https://alas.aws.amazon.com/AL2023/ALAS-2023-343.html for an example of this for libxml2.

# - OpenSSL
# Needed by:
# - curl
ENV VERSION_LIBSSH2=1.11.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 1.10.0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security issues in the native version?

# - libxml2
# Needed by:
# - curl
ENV VERSION_NGHTTP2=1.56.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 1.55.1

# # - libnghttp2
# # Needed by:
# # - php
ENV VERSION_CURL=8.3.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 8.2.1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't be possible, once we need to start building curl with http/3 enabled.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we do move curl forward in Amazon Linux fairly regularly, often as part of patching security issues.

If there's anything specifically not enabled in our curl build that you need, please file an issue over at https://github.com/amazonlinux/amazon-linux-2023/issues so we can look into it.

Note that in AL2023 there is both a curl and curl-minimal package (and the associated libcurl and libcurl-minimal). The minimal ones are installed by default, and we try and capture the 99% of use cases there (e.g. don't build in ldap:// support and the like).

# https://github.com/nih-at/libzip/releases
# Needed by:
# - php
ENV VERSION_ZIP=1.10.1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 1.7.3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This native version is wayyyyy too old.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😄 thanks for all the feedback, happy I opened this this is useful (to know we may not want to drop all the compilation we are doing)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open an issue up at https://github.com/amazonlinux/amazon-linux-2023/issues detailing the issues with the older version that would require an update?

# - OpenSSL
# Needed by:
# - php
ENV VERSION_POSTGRES=15.4
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 15.0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is over a year old.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you cut us an issue at https://github.com/amazonlinux/amazon-linux-2023/issues detailing what from the client side you need updated?

# https://github.com/kkos/oniguruma/releases
# Needed by:
# - php mbstring
ENV VERSION_ONIG=6.9.8
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native version: 6.9.7.1

@GrahamCampbell
Copy link
Contributor

AL2023 is eventually going to happen

I am actually not so sure. 😆

@mnapoli
Copy link
Member Author

mnapoli commented Sep 27, 2023

@GrahamCampbell that wasn't long :) aws/aws-lambda-base-images#92

@GrahamCampbell
Copy link
Contributor

Ha, it's not available on the Lambda APIs and cloudformation yet. ;)

-DCMAKE_BUILD_TYPE=RELEASE
RUN cmake --build . --target install


###############################################################################
# LIBSODIUM
# https://github.com/jedisct1/libsodium/releases

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building libsodium shouldn't be required on AL2023 as there's a version in the repositories as of AL2023.2 (see https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.2.20230920.html )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it is not the latest version.

php-81/Dockerfile Outdated Show resolved Hide resolved
@mnapoli
Copy link
Member Author

mnapoli commented Oct 10, 2023

Thanks so much for jumping in the discussion @stewartsmith.

There is some balance that we need to find.

Would it be reasonable to apply the following logic:

  • When we need specific features (random example: HTTP3 support in curl) that are not present in AL2023, we recompile the libraries
  • The rest of the time, we rely on AL2023 versions for CVEs and security fixes

Is that reasonable enough? Can we assume that all CVEs are patched in AL2023 eventually? (I understand it may take a few hours/days depending on the issue)

The benefits of that approach would be:

  • our compilation scripts are simpler
  • our layers might potentially be lighter (faster cold starts)
  • security fixes are delivered in the Lambda OS, not via a new version of the Bref layers -> for most Bref users, that means that security fixes are actually applied faster, because most Bref users don't necessarily update Bref layers every week or so (meaning they are actually using older unpatched system libraries provided by Bref).

WDYT?

@stewartsmith
Copy link

Building anything from source and replacing an OS component with it carries various risks. Naturally it means you have to keep it updated, but there's also the nuance of building it with the same hardening and optimisations as the one in the OS, and ensuring ABI compatibility with the one in the OS. It's not something we recommend doing.

It is (of course) something which is possible to do, as it's basically what we do all the time when building updates for the OS. From a "does the Amazon Linux team support this" standpoint, replacing core system libraries falls into "can you reproduce the issue without that change" territory - i.e. a freedom that Free and Open Source Software gives you, but the vendor doesn't necessarily provide support for if you pick up the (metaphorical or literal) phone.

I would recommend building RPMs instead of manually from source tarballs though, as that way you capture all the hardening and optimization flags that are added in the RPM macros.

In Amazon Linux, security is job zero. We do fix security issues of all severity, and aim to do so not just in the most recent major version of the OS. The places where don't patch something basically boils down to risk outweighing the benefit - such as Low CVEs that are a quite invasive fix that we can't reasonably bring in without creating other, more severe issues.

@mnapoli
Copy link
Member Author

mnapoli commented Oct 17, 2023

Thanks a lot @stewartsmith for chiming in. I've taken some time to think about it, and also discuss this with a few Bref users. For example I discussed this topic with a bank running on Bref (i.e. they have huge security constraints).

The feedback was quite simple: if we build our own versions, we (Bref project) are responsible for the security related to those libraries. If we don't, AWS is.

To summarize:

  • Bref users looking for strong security guarantees will usually prefer to delegate as much as possible to AWS (https://docs.aws.amazon.com/whitepapers/latest/security-overview-aws-lambda/the-shared-responsibility-model.html). This allows ticking a lot of boxes.
  • Additionally, most Bref users will not update the runtime versions frequently enough, leaving them more exposed to CVEs and security issues (in the case where we build libraries). By using system libraries instead, AWS is able to patch the underlying runtime automatically, which gives a more secure environment.

That leads me to conclude that we should aim to compile as few libraries as possible. This will benefit both extremes of the spectrum: users with strong security requirements and those that don't update frequently (i.e. care less about security, if I can say so).

The downside is that we have no way to control the frequency/speed at which AWS patches security issues. This basically means putting more trust into AWS over Bref regarding security, and I think that is a reasonable choice 😅


This PR is still WIP, but I'll be slowly updating it to reflect that. I.e. the build scripts should be simpler with AL2023 than AL2.

Also since we'll want to support both, I'll probably reorganize the code entirely. We will also likely not support PHP 8.0 with AL2023.

@@ -11,23 +11,14 @@ ARG VERSION_PHP=8.0.30
# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
# AWS provides a Docker image that we use here:
# https://github.com/amazonlinux/container-images/tree/amzn2
FROM public.ecr.aws/lambda/provided:al2-${IMAGE_VERSION_SUFFIX} as build-environment
FROM public.ecr.aws/lambda/provided:al2023-preview-${IMAGE_VERSION_SUFFIX} as build-environment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to pin this to al2023.2, rather than just al2023?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm, not really. just al2023-preview.2023.10.17.17

@shiki-sou
Copy link

Hello,
Thank you for your continuous work on the Bref project.
It seems this ticket has been postponed for a little bit while.
I'm interested in how long Amazon Linux 2023 will be supported.
And is there any key milestones or timelines related to this support?

@hrzbrg
Copy link

hrzbrg commented May 15, 2024

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc.
See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

@shiki-sou
Copy link

Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc. See their release cadence: https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html

Thank you for answering my question.
Sorry about not asking question clearly.

I want to know the schedual that brefphp/aws-lambda-layers support Amazon Linux 2023 which this ticket was working on (If I'm not misunderstanding).
Because I found that that brefphp/aws-lambda-layers only suppot amazon linux 2(provided.al2).
See the capture below (I get this capture from Lambda function which I deployed)
image

@mnapoli
Copy link
Member Author

mnapoli commented May 17, 2024

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

@shiki-sou
Copy link

Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍

Thank you for your reply.

@kevincerro
Copy link
Contributor

kevincerro commented Jun 28, 2024

Hi!! 🚀

Happy to help with this.
There is something to do? @mnapoli

@mnapoli
Copy link
Member Author

mnapoli commented Jul 4, 2024

There was a discussion in Slack but to update here: this is in my TODO, this is related to many other changes with Bref v3 that's why it's progressing seemingly slowly.

@jiw-mh
Copy link

jiw-mh commented Jul 25, 2024

We run nodejs on the same machine as bref and modern versions of nodejs do not compile on al2 so we are tied to an old nodejs version. An al2023 update would allow us to update nodejs as well.

# Conflicts:
#	php-83/Dockerfile
#	tests/test_2_extensions.php
#	utils/lib-copy/libs-arm.txt
#	utils/lib-copy/libs-x86.txt
Copy link
Contributor

@GrahamCampbell GrahamCampbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want newer sqlite, from source.

@mnapoli
Copy link
Member Author

mnapoli commented Oct 24, 2024

@GrahamCampbell why? Are we missing a feature with the version shipped in AL2023?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants