AWS Deployments are failing with the message "no module named 'cryptography'"

[sjanakirampowercor]

Describe the bug

We have BuildKite as our CI/CD pipeline to deploy applications on AWS infrastructure.

We recently updated our BuildKite agent to a new EC2 AMI to adhere to Node v18 runtime.

AMI Details:
Architecture
x86_64
Instance type
t2.medium

This was working absolutely fine until Wednesday 4th of October 2023.
We have a Bootstrap script that installs the AWS CLI along with a bunch of other things whenever the BuildKite EC2 instance is spun up for deployment.

Here's a snippet of the script:

#----------------------------------------------------#

Update aws-cli

#----------------------------------------------------#
yum -y install aws-cli

#----------------------------------------------------#

Install AWS CLI tools

#----------------------------------------------------#
pip install awsebcli --upgrade --ignore-installed
python3 -m pip install aws-sam-cli

The above commands are causing issues and resulting in unsuccessful installation of the AWS CLI. Here's what we're getting from the system logs:

<13>Oct 6 21:38:58 user-data: awscli 2.9.19 requires cryptography<=38.0.5,>=3.3.2, but you have cryptography 41.0.4 which is incompatible.
<13>Oct 6 21:38:58 user-data: awscli 2.9.19 requires ruamel.yaml<=0.17.21,>=0.15.0, but you have ruamel-yaml 0.17.35 which is incompatible.

All of a sudden the deployments are no longer working and causing serious blockers. We can confirm that we haven't changed anything on the Bootstrap script. We suspect this could be an issue with the version upgrade released yesterday.

We even tried to install a specific version of cryptography as per the above error message, but the CLI is still failing to install successfully.

Expected Behavior

The AWS CLI deployments must work fine without any issues.

Current Behavior

All deployments are failing since AWS CLI is not being installed successfully and the error message "no module named 'cryptography'" pops up.

Reproduction Steps

1. Trigger a deployment via BuildKite to deploy the application onto AWS EC2 instances.
2. The BuildKite Bootstrap script installs the AWS CLI and relevant dependencies.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.9

Environment details (OS name and version, etc.)

AMI Details: Architecture x86_64 Instance type t2.medium

[sjanakirampowercor]

Update: We got to the bottom of this. We're installing AWS CLI and AWS SAM CLI (https://pypi.org/project/aws-sam-cli/1.98.0/).

The order in which we install them are: AWS CLI followed by AWS SAM CLI.

The SAM CLI that came out with a recent release on the 4th of October 2023, tries installing a version of cryptography library 41.0, whereas the AWS CLI requires a cryptography version of <38.0. This causes issues with the installation.

As a workaround we ended up locking down the version of SAM CLI. But I still feel this needs to be fixed either at AWS CLI or SAM CLI as we can't have the best of both worlds by using the latest versions at the same time.

[aBurmeseDev]

Hi @sjanakirampowercor - thanks for reaching out and sorry to hear you're having trouble.

It looks like you're installing AWS CLI v2 via yum which we do not officially support, unfortunately. Here's our official documentation on how to install/updates AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.

For cryptography and other deps, we have a tracking issue open for updating the version ranges of several dependencies here: #5943.

If you're still having trouble after installing AWS CLI with one of the methods from the official doc linked above, please let us know and we'd be happy to investigate further.

[sjanakirampowercor]

@aBurmeseDev Thank you for reaching out! We noticed the issue with yum and migrated away from it, but that didn't resolve the issue either. The only workaround right now is to lock down the SAM CLI version to ensure cryptography does not get uninstalled and re-installed. I did see the other open issue that you've linked above before raising this one. I hope once that is released, we wouldn't face issues anymore. Happy for this case to be closed as the above issue would potentially solve our workaround.

[aBurmeseDev]

Thanks for letting me know, feel free to reach out if you have any other questions! I'll go ahead and close this one out.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.