Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Dependencies] Upgrade Python version to 3.12 and removed pinning for setuptools #6465

Merged
merged 4 commits into from
Oct 14, 2024
Merged

[Dependencies] Upgrade Python version to 3.12 and removed pinning for setuptools #6465

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b7cf9a
Reapply "Upgrade Lambda Runtime for Pcluster Lambda Layer"
hanwen-cluster Oct 9, 2024
ef7327a
Reapply "Pin jsonschema for Lambda Layer to version 4.17.3"
hanwen-cluster Oct 11, 2024
5c63eba
Revert "Pin setuptools version to be less than 70.0.0"
hanwen-cluster Oct 10, 2024
4a0600a
[Documentation] Fix changelog entries for version 3.11.1.
gmarciani Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ CHANGELOG

**CHANGES**
- Pyxis is now disabled by default and it must be manually enabled on the head node.
- Upgrade Python runtime to version 3.12 in ParallelCluster Lambda Layer.
- Remove version pinning for setuptools to version prior to 70.0.0.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, we did not document CLI Python dependencies upgrade in ChangeLog. So this line is optional

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth it to document these dependencies because:

  1. Changing the PC Lambda Layer runtime is equivalent to making pcluster compatible with a newer version of Python. When old Lambda runtimes are used, users may receive warnings; this upgrade may solve these warnings and it's useful to keep track of this improvement.
  2. setuptools: this package affects the pcluster installation, so if the user faces issues in installing pcluster it may also be because of changes in the new version of setuptools.


**BUG FIXES**
- Fix an issue in the way we configure the Pyxis Slurm plugin in ParallelCluster that can lead to job submission failures.
Expand Down
4 changes: 2 additions & 2 deletions api/infrastructure/parallelcluster-api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ Resources:
- parallelcluster/${Version}/layers/aws-parallelcluster/lambda-layer.zip
- { Version: !FindInMap [ParallelCluster, Constants, Version]}
CompatibleRuntimes:
- python3.9
- python3.12

# We need to define three AWS::Serverless::Api due to an issue with the handling of AWS::NoValue
# See related GitHub issue: https://github.com/aws/serverless-application-model/issues/1435
Expand Down Expand Up @@ -294,7 +294,7 @@ Resources:
Value: api
- Key: 'parallelcluster:version'
Value: !FindInMap [ParallelCluster, Constants, Version]
Runtime: python3.9
Runtime: python3.12
Handler: pcluster.api.awslambda.entrypoint.lambda_handler
Layers:
- !Ref PclusterLayer
Expand Down
2 changes: 1 addition & 1 deletion awsbatch-cli/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def readme():

VERSION = "1.4.0"
REQUIRES = [
"setuptools<70.0.0",
"setuptools",
"boto3>=1.16.14",
"tabulate>=0.8.8,<=0.8.10",
]
Expand Down
2 changes: 1 addition & 1 deletion cli/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def readme():
VERSION = "3.11.1"
CDK_VERSION = "1.164"
REQUIRES = [
"setuptools<70.0.0",
"setuptools",
"boto3>=1.16.14",
"tabulate>=0.8.8,<=0.8.10",
"PyYAML>=5.3.1,!=5.4",
Expand Down
6 changes: 3 additions & 3 deletions cloudformation/custom_resource/cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Resources:
- parallelcluster/${Version}/layers/aws-parallelcluster/lambda-layer.zip
- { Version: !FindInMap [ParallelCluster, Constants, Version] }
CompatibleRuntimes:
- python3.9
- python3.12

PclusterPolicies:
Condition: UsePCPolicies
Expand Down Expand Up @@ -341,7 +341,7 @@ Resources:
helper(event, context)

Handler: index.handler
Runtime: python3.9
Runtime: python3.12
Role: !If [CustomRoleCondition, !Ref CustomLambdaRole, !GetAtt PclusterLambdaRole.Arn]
Layers:
- !Ref PclusterLayer
Expand Down Expand Up @@ -395,7 +395,7 @@ Resources:
reason = str(e)
cfnresponse.send(event, context, response_status, {}, event.get('PhysicalResourceId', 'CleanupS3bucketCustomResource'), reason)
Handler: index.handler
Runtime: python3.9
Runtime: python3.12
Role: !If [CustomRoleCondition, !Ref CustomLambdaRole, !GetAtt PclusterLambdaRole.Arn]

CleanupS3bucketCustomResource:
Expand Down
2 changes: 1 addition & 1 deletion cloudformation/external-slurmdbd/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
setuptools<70.0.0
setuptools
aws-cdk-lib~=2.105
constructs>=10.0.0,<11.0.0
6 changes: 4 additions & 2 deletions tests/integration-tests/conftest_resource_bucket.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
from tests.common.utils import get_installed_parallelcluster_version

logger = logging.getLogger()
NODE_VERSION = "v16.19.0" # maintenance version compatible with alinux2's GLIBC
NODE_VERSION = "v18.20.3"


def install_pc(basepath, pc_version):
Expand All @@ -40,7 +40,9 @@ def install_pc(basepath, pc_version):
cli_dir = root / "cli"
try:
logger.info("installing ParallelCluster packages...")
subprocess.check_call([sys.executable, "-m", "pip", "install", f"{cli_dir}[awslambda]", "-t", tempdir])
subprocess.check_call(
[sys.executable, "-m", "pip", "install", "jsonschema==4.17.3", f"{cli_dir}[awslambda]", "-t", tempdir]
)
# The following are provided by the lambda runtime
shutil.rmtree(tempdir / "botocore")
shutil.rmtree(tempdir / "boto3")
Expand Down
Loading