Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[macOS] OpenSSL 1.1 will be removed and OpenSSL 3 will be the default for all macOS images from November 4 #10817

Open
7 of 14 tasks
sarathrajsrinivasan opened this issue Oct 18, 2024 · 18 comments

Comments

@sarathrajsrinivasan
Copy link
Contributor

Breaking changes

OpenSSL 1.1 will be removed from all macOS images and the default version will be switched to OpenSSL 3.

Target date

Image deployment will start on November 4 and will take 2-3 days.

The motivation for the changes

OpenSSL 1.1 has reached its end-of-life (EOL) and is no longer supported. Due to its deprecation, Homebrew has removed it from its available packages.

Possible impact

If your library/project is not compatible with OpenSSL 3 and you build them based on default OpenSSL version - your workflow will be broken.

Platforms affected

  • Azure DevOps
  • GitHub Actions

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • macOS 12
  • macOS 13
  • macOS 13 Arm64
  • macOS 14
  • macOS 14 Arm64
  • macOS 15
  • macOS 15 Arm64
  • Windows Server 2019
  • Windows Server 2022

Mitigation ways

Use the below step in your workflow to download and install OpenSSL 1.1 manually

    - name : Install  OpenSSL 1.1 manually
      run : |
        wget https://www.openssl.org/source/openssl-1.1.1w.tar.gz
        tar -xvf openssl-1.1.1w.tar.gz
        cd openssl-1.1.1w
        ./config --prefix=/usr/local/openssl
        make
        sudo make install
        export PATH=/usr/local/openssl/bin:$PATH
@eregon
Copy link

eregon commented Nov 7, 2024

This means currently some runs on macos-14 have [email protected] and some don't:
https://github.com/eregon/setup-ruby/actions/runs/11724213556/job/32657603152

This is quite problematic because if something is built on macos-14 on a run with [email protected] (and the software links to that) it can fail when that's later used in another run as [email protected] /opt/homebrew/opt/[email protected]/lib/libssl.1.1.dylib is gone.

Basically, anything that ever linked to [email protected] on macos-* will be broken by this change (until it's recompiled against openssl@3).

graalvmbot pushed a commit to oracle/truffleruby that referenced this issue Nov 7, 2024
* Should help when 1.1 is removed like in actions/runner-images#10817
* 1.1 is EOL
@jorupp
Copy link

jorupp commented Nov 7, 2024

Is there a way to configure our build so we only run on the OpenSSL@3-equipped images? We can't add the -legacy flag we need for OpenSSL@3 (called within InstallAppleCertificate@2's pre-job) because it breaks when the build runs on an [email protected]-equipped image, and we can't manually install [email protected] via a script because the call to openssl is in InstallAppleCertificate@2's pre-job.

@jorupp
Copy link

jorupp commented Nov 7, 2024

We worked around it with InstallAppleCertificate@2 by putting two copies of the task in our pipeline - one with opensslPkcsArgs: -legacy and one without, and with continueWithError: true on both so the one that fails won't fail our build.

@mikeharder
Copy link

Could this change be breaking actions/setup-python on Mac?

#10812 (comment)

@altendky
Copy link

altendky commented Nov 8, 2024

In this test run https://github.com/Chia-Network/chia-blockchain/actions/runs/11745259139?pr=18844 with 30x each of macOS ARM/Intel and Python version 3.9/3.10/3.11/3.12, only 3.10 on Intel failed. The passing jobs were run on either 20241008.186 or 20241023.237. The failing jobs were on 20241107.313.

@mikeharder
Copy link

In this test run https://github.com/Chia-Network/chia-blockchain/actions/runs/11745259139?pr=18844 with 30x each of macOS ARM/Intel and Python version 3.9/3.10/3.11/3.12, only 3.10 on Intel failed. The passing jobs were run on either 20241008.186 or 20241023.237. The failing jobs were on 20241107.313.

I think we might be seeing a combination of both #10817 and #10812. In your test run, Python 3.10 was found in the cache on macOS ARM64, so it passes. But if you land on an agent without Python 3.10 in the cache, it will try to download and install, but fails with the SSL error.

@fengga
Copy link

fengga commented Nov 8, 2024

Is there a way to configure our build so we only run on the OpenSSL@3-equipped images? We can't add the -legacy flag we need for OpenSSL@3 (called within InstallAppleCertificate@2's pre-job) because it breaks when the build runs on an [email protected]-equipped image, and we can't manually install [email protected] via a script because the call to openssl is in InstallAppleCertificate@2's pre-job.

I'm facing the same problem and if rollout to [email protected] all the InstallAppleCertificate@2 task will be broken.
Even if I use a cert signed by [email protected] there is still a problem:

InstallAppleCertificate@2 will generate one command:

security import ***/MSALCPPAppleBuildCertificate_fgh.p12 -P *** -A -t cert -f pkcs12 -k ***/ios_signing_temp.keychain

which requires the MSALCPPAppleBuildCertificate_fgh.p12 signed by [email protected]

https://identitydivision.visualstudio.com/IDDP/_build/results?buildId=1385795&view=logs&j=9c650f88-aede-5522-025a-32fa3c2465b6&t=a74cb428-1f82-4e39-8054-c06bdb9b0377&l=20

@sarathrajsrinivasan
Copy link
Contributor Author

Hi All,

We are currently reverting the change and keeping OpenSSL 1.1 in our images. It will be the default version for now.
Will keep you posted with the updates.

@dalemyers
Copy link

@sarathrajsrinivasan Why was this reverted? After having spent time adjusting our flows to make it work with v3, we now need to undo that work, causing more disruption. This is likely the case for many others.

@eregon
Copy link

eregon commented Nov 11, 2024

@dalemyers There is no need to undo that work, openssl@3 is available before & after that change.

@dalemyers
Copy link

dalemyers commented Nov 11, 2024

@dalemyers There is no need to undo that work, openssl@3 is available before & after that change.

It is, but it's not the default, and various tasks expect one version or the other, so they need to be adjusted. Anything we control calling openssl directly is fine.

@gabsamples6
Copy link

We are still facing this issues and we are really stuck, is the fix/revert going to be applied soon? thanks

@jorupp
Copy link

jorupp commented Nov 19, 2024

We are still facing this issues and we are really stuck, is the fix/revert going to be applied soon? thanks

Are you seeing v1.1.1 or v3 right now? We've been running ours with both versions of the task that's having an issue and using continueWithError: true to ignore the failing one. Every run this week has had the v1.1.1-style call succeed and the v3-style call fail.

@surendrapatel-rak
Copy link

We are still facing this issues and we are really stuck, is the fix/revert going to be applied soon? thanks

Are you seeing v1.1.1 or v3 right now? We've been running ours with both versions of the task that's having an issue and using continueWithError: true to ignore the failing one. Every run this week has had the v1.1.1-style call succeed and the v3-style call fail.

Is it not possible to skip the task (not needed) based on the agent macOS version check?

@gabsamples6
Copy link

gabsamples6 commented Nov 20, 2024

I have just tried on using both and both failed. I added "continueOnError" and legacy, This is just mad now - what are we going to do? Can someone post a full workaround if any? We cannot deploy an app to prod because of this

    - task: InstallAppleCertificate@2
      displayName: 'Install an Apple certificate ${{ parameters.appleCertificateP12 }} '
      inputs:
        certSecureFile: ${{ parameters.appleCertificateP12 }}
        certPwd: '${{ parameters.p12password }}'
        keychain: 'temp'
        opensslPkcsArgs: '-legacy'       USING LEGACY HERE
     continueOnError: true
      
    - task: InstallAppleCertificate@2
      displayName: 'Install an Apple certificate ${{ parameters.appleCertificateP12 }} '
      inputs:
        certSecureFile: ${{ parameters.appleCertificateP12 }}
        certPwd: '${{ parameters.p12password }}'
        keychain: 'temp'
      continueOnError: true
	  

@d-philipson
Copy link

It's currently potluck if you get a runner image that actually works. 20241022.244 works while 20241106.316 fails. I just have to keep running the build until it eventually gets the older image and completes. It's absolutely ridiculous.

@jorupp
Copy link

jorupp commented Nov 20, 2024

I'm seeing both 20241022.254 and 20241118.366, but both are having the task with -legacy fail and the one without succeed (which works for our build thanks to continueOnError: true). /usr/local/bin/openssl version is saying OpenSSL 1.1.1w 11 Sep 2023 for both.

@bwburch
Copy link

bwburch commented Nov 20, 2024

@d-philipson yes I 100% concur this is ridiculous

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

13 participants