-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Preliminary work to compile on Amazon Linux 2023 #122
base: main
Are you sure you want to change the base?
Conversation
php-80/Dockerfile
Outdated
# - libzip: minimum required CMAKE version 3.0. | ||
RUN LD_LIBRARY_PATH= yum install -y cmake3 | ||
# Override the default `cmake` | ||
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmake3 is installed by default
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is due to AL2 having both cmake
and cmake3
packages to ensure that software needing cmake
that didn't yet work with cmake3
could continue to be built.
For AL2023, it's all cmake
version 3.
php-81/Dockerfile
Outdated
# - curl | ||
# - php | ||
RUN yum install -y perl-IPC-Cmd | ||
ENV VERSION_OPENSSL=3.0.10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 3.0.8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Security issues in the native version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried asking around but have no answers yet. Do you know if the versions compiled in AL might have some security fixes applied?
I'm just surprised AL would ship with security issues, how do all customers do? I wouldn't expect everyone to compile everything from source, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We backport security fixes to the OpenSSL versions that we ship. See https://alas.aws.amazon.com/alas2023.html for updates.
For example: https://alas.aws.amazon.com/AL2023/ALAS-2023-306.html and https://alas.aws.amazon.com/AL2023/ALAS-2023-222.html are some recent ones.
If there's anything from newer OpenSSL that you need, please do let us know and submit an issue to https://github.com/amazonlinux/amazon-linux-2023/ for us to look at.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need a version that supports quic possibly, for http/3 in curl, also. I am not sure how long it takes AWS to apply security fixes. I like the flexibility that keeping this gives us to easily re-compile, or for me to fork and re-compile to get something patched immediately.
php-81/Dockerfile
Outdated
# Needed by: | ||
# - php | ||
# - libnghttp2 | ||
ENV VERSION_XML2=2.11.5 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 2.10.4
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Security issues in the native version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do backport security fixes to the version we ship in Amazon Linux. See https://alas.aws.amazon.com/AL2023/ALAS-2023-343.html for an example of this for libxml2
.
php-81/Dockerfile
Outdated
# - OpenSSL | ||
# Needed by: | ||
# - curl | ||
ENV VERSION_LIBSSH2=1.11.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 1.10.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Security issues in the native version?
php-81/Dockerfile
Outdated
# - libxml2 | ||
# Needed by: | ||
# - curl | ||
ENV VERSION_NGHTTP2=1.56.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 1.55.1
php-81/Dockerfile
Outdated
# # - libnghttp2 | ||
# # Needed by: | ||
# # - php | ||
ENV VERSION_CURL=8.3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 8.2.1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't be possible, once we need to start building curl with http/3 enabled.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that we do move curl
forward in Amazon Linux fairly regularly, often as part of patching security issues.
If there's anything specifically not enabled in our curl
build that you need, please file an issue over at https://github.com/amazonlinux/amazon-linux-2023/issues so we can look into it.
Note that in AL2023 there is both a curl
and curl-minimal
package (and the associated libcurl
and libcurl-minimal
). The minimal ones are installed by default, and we try and capture the 99% of use cases there (e.g. don't build in ldap://
support and the like).
php-81/Dockerfile
Outdated
# https://github.com/nih-at/libzip/releases | ||
# Needed by: | ||
# - php | ||
ENV VERSION_ZIP=1.10.1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 1.7.3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This native version is wayyyyy too old.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😄 thanks for all the feedback, happy I opened this this is useful (to know we may not want to drop all the compilation we are doing)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you open an issue up at https://github.com/amazonlinux/amazon-linux-2023/issues detailing the issues with the older version that would require an update?
php-81/Dockerfile
Outdated
# - OpenSSL | ||
# Needed by: | ||
# - php | ||
ENV VERSION_POSTGRES=15.4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 15.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is over a year old.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you cut us an issue at https://github.com/amazonlinux/amazon-linux-2023/issues detailing what from the client side you need updated?
php-81/Dockerfile
Outdated
# https://github.com/kkos/oniguruma/releases | ||
# Needed by: | ||
# - php mbstring | ||
ENV VERSION_ONIG=6.9.8 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Native version: 6.9.7.1
I am actually not so sure. 😆 |
@GrahamCampbell that wasn't long :) aws/aws-lambda-base-images#92 |
Ha, it's not available on the Lambda APIs and cloudformation yet. ;) |
php-80/Dockerfile
Outdated
-DCMAKE_BUILD_TYPE=RELEASE | ||
RUN cmake --build . --target install | ||
|
||
|
||
############################################################################### | ||
# LIBSODIUM | ||
# https://github.com/jedisct1/libsodium/releases |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Building libsodium
shouldn't be required on AL2023 as there's a version in the repositories as of AL2023.2 (see https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.2.20230920.html )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, it is not the latest version.
Thanks so much for jumping in the discussion @stewartsmith. There is some balance that we need to find. Would it be reasonable to apply the following logic:
Is that reasonable enough? Can we assume that all CVEs are patched in AL2023 eventually? (I understand it may take a few hours/days depending on the issue) The benefits of that approach would be:
WDYT? |
Building anything from source and replacing an OS component with it carries various risks. Naturally it means you have to keep it updated, but there's also the nuance of building it with the same hardening and optimisations as the one in the OS, and ensuring ABI compatibility with the one in the OS. It's not something we recommend doing. It is (of course) something which is possible to do, as it's basically what we do all the time when building updates for the OS. From a "does the Amazon Linux team support this" standpoint, replacing core system libraries falls into "can you reproduce the issue without that change" territory - i.e. a freedom that Free and Open Source Software gives you, but the vendor doesn't necessarily provide support for if you pick up the (metaphorical or literal) phone. I would recommend building RPMs instead of manually from source tarballs though, as that way you capture all the hardening and optimization flags that are added in the RPM macros. In Amazon Linux, security is job zero. We do fix security issues of all severity, and aim to do so not just in the most recent major version of the OS. The places where don't patch something basically boils down to risk outweighing the benefit - such as Low CVEs that are a quite invasive fix that we can't reasonably bring in without creating other, more severe issues. |
Thanks a lot @stewartsmith for chiming in. I've taken some time to think about it, and also discuss this with a few Bref users. For example I discussed this topic with a bank running on Bref (i.e. they have huge security constraints). The feedback was quite simple: if we build our own versions, we (Bref project) are responsible for the security related to those libraries. If we don't, AWS is. To summarize:
That leads me to conclude that we should aim to compile as few libraries as possible. This will benefit both extremes of the spectrum: users with strong security requirements and those that don't update frequently (i.e. care less about security, if I can say so). The downside is that we have no way to control the frequency/speed at which AWS patches security issues. This basically means putting more trust into AWS over Bref regarding security, and I think that is a reasonable choice 😅 This PR is still WIP, but I'll be slowly updating it to reflect that. I.e. the build scripts should be simpler with AL2023 than AL2. Also since we'll want to support both, I'll probably reorganize the code entirely. We will also likely not support PHP 8.0 with AL2023. |
php-80/Dockerfile
Outdated
@@ -11,23 +11,14 @@ ARG VERSION_PHP=8.0.30 | |||
# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html | |||
# AWS provides a Docker image that we use here: | |||
# https://github.com/amazonlinux/container-images/tree/amzn2 | |||
FROM public.ecr.aws/lambda/provided:al2-${IMAGE_VERSION_SUFFIX} as build-environment | |||
FROM public.ecr.aws/lambda/provided:al2023-preview-${IMAGE_VERSION_SUFFIX} as build-environment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a way to pin this to al2023.2, rather than just al2023?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmmm, not really. just al2023-preview.2023.10.17.17
Hello, |
Amazon Linux 2023 will be supported officially until 2027. However every 2 years Amazon will release a new major version. So in 2025 there will be the release of Amazon Linux 2025, which in turn will be supported until 2029 etc. |
Thank you for answering my question. I want to know the schedual that brefphp/aws-lambda-layers support Amazon Linux 2023 which this ticket was working on (If I'm not misunderstanding). |
Hi! I have made progress on this offline (not pushed here). This is definitely in my TODO list, though not the most urgent because AL2 works well and is supported by Amazon. But this is not something I have forgotten about or abandoned 👍 |
Thank you for your reply. |
Hi!! 🚀 Happy to help with this. |
There was a discussion in Slack but to update here: this is in my TODO, this is related to many other changes with Bref v3 that's why it's progressing seemingly slowly. |
We run nodejs on the same machine as bref and modern versions of nodejs do not compile on al2 so we are tied to an old nodejs version. An al2023 update would allow us to update nodejs as well. |
# Conflicts: # php-83/Dockerfile # tests/test_2_extensions.php # utils/lib-copy/libs-arm.txt # utils/lib-copy/libs-x86.txt
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably want newer sqlite, from source.
@GrahamCampbell why? Are we missing a feature with the version shipped in AL2023? |
AL2023 is eventually going to happen. Out of curiosity, I drafted this PR trying to build our layers for AL2023.
It removes a lot of custom system libraries we compile because AL2023 contains more recent versions: https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.1.html
We'll need to reevaluate when we actually get AL2023 Lambda images (right now we have the generic AL2023, not specific to Lambda), but it is looking good.
I'm opening this PR to gather early feedback and see if I'm missing something.