Configure the Compute Instances
@@ -226,13 +242,204 @@ Install Cloud Development Ki
Note that the version of aws-cdk changes frequently.
The version that has been tested is in the CDK_VERSION variable in the install script.
The install script will try to install the prerequisites if they aren't already installed.
+Security Groups for Login Nodes
+If you want to allow instances like remote desktops to use the cluster directly, you must define
+three security groups that allow connections between the instance, the Slurm head node, and the Slurm compute nodes.
+We call the instance that is connecting to the Slurm cluster a login node or a submitter instance.
+I'll call the three security groups the following names, but they can be whatever you want.
+
+- SlurmSubmitterSG
+- SlurmHeadNodeSG
+- SlurmComputeNodeSG
+
+Slurm Submitter Security Group
+The SlurmSubmitterSG will be attached to your login nodes, such as your virtual desktops.
+It needs at least the following inbound rules:
+
+
+
+| Type |
+Port range |
+Source |
+Description |
+
+
+
+
+| TCP |
+1024-65535 |
+SlurmHeadNodeSG |
+SlurmHeadNode ephemeral |
+
+
+| TCP |
+1024-65535 |
+SlurmComputeNodeSG |
+SlurmComputeNode ephemeral |
+
+
+| TCP |
+6000-7024 |
+SlurmComputeNodeSG |
+SlurmComputeNode X11 |
+
+
+
+It needs the following outbound rules.
+
+
+
+| Type |
+Port range |
+Destination |
+Description |
+
+
+
+
+| TCP |
+2049 |
+SlurmHeadNodeSG |
+SlurmHeadNode NFS |
+
+
+| TCP |
+6818 |
+SlurmComputeNodeSG |
+SlurmComputeNode slurmd |
+
+
+| TCP |
+6819 |
+SlurmHeadNodeSG |
+SlurmHeadNode slurmdbd |
+
+
+| TCP |
+6820-6829 |
+SlurmHeadNodeSG |
+SlurmHeadNode slurmctld |
+
+
+| TCP |
+6830 |
+SlurmHeadNodeSG |
+SlurmHeadNode slurmrestd |
+
+
+
+Slurm Head Node Security Group
+The SlurmHeadNodeSG will be specified in your configuration file for the slurm/SlurmCtl/AdditionalSecurityGroups parameter.
+It needs at least the following inbound rules:
+
+
+
+| Type |
+Port range |
+Source |
+Description |
+
+
+
+
+| TCP |
+2049 |
+SlurmSubmitterSG |
+SlurmSubmitter NFS |
+
+
+| TCP |
+6819 |
+SlurmSubmitterSG |
+SlurmSubmitter slurmdbd |
+
+
+| TCP |
+6820-6829 |
+SlurmSubmitterSG |
+SlurmSubmitter slurmctld |
+
+
+| TCP |
+6830 |
+SlurmSubmitterSG |
+SlurmSubmitter slurmrestd |
+
+
+
+It needs the following outbound rules.
+
+
+
+| Type |
+Port range |
+Destination |
+Description |
+
+
+
+
+| TCP |
+1024-65535 |
+SlurmSubmitterSG |
+SlurmSubmitter ephemeral |
+
+
+
+Slurm Compute Node Security Group
+The SlurmComputeNodeSG will be specified in your configuration file for the slurm/InstanceConfig/AdditionalSecurityGroups parameter.
+It needs at least the following inbound rules:
+
+
+
+| Type |
+Port range |
+Source |
+Description |
+
+
+
+
+| TCP |
+6818 |
+SlurmSubmitterSG |
+SlurmSubmitter slurmd |
+
+
+
+It needs the following outbound rules.
+
+
+
+| Type |
+Port range |
+Destination |
+Description |
+
+
+
+
+| TCP |
+1024-65535 |
+SlurmSubmitterSG |
+SlurmSubmitter ephemeral |
+
+
+| TCP |
+6000-7024 |
+SlurmSubmitterSG |
+SlurmSubmitter X11 |
+
+
+
Create Configuration File
Before you deploy a cluster you need to create a configuration file.
A default configuration file is found in source/resources/config/default_config.yml.
You should create a new config file and update the parameters for your cluster.
Ideally you should version control this file so you can keep track of changes.
The schema for the config file along with its default values can be found in source/cdk/config_schema.py.
-The schema is defined in python, but the actual config file should be in yaml format.
+The schema is defined in python, but the actual config file should be in yaml format.
+See Configuration File Format for documentation on all of the parameters.
The following are key parameters that you will need to update.
If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option.
You should save your selections in the config file.
@@ -277,12 +484,6 @@ Create Configuration File
None |
-| slurm/SubmitterSecurityGroupIds |
-Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. |
-sg-* |
-None |
-
-
| ErrorSnsTopicArn |
ARN of an SNS topic that will be notified of errors |
arn:aws:sns:{{region}}:{AccountId}:{TopicName} |
diff --git a/index.html b/index.html
index 4a78c8f0..3050996c 100644
--- a/index.html
+++ b/index.html
@@ -309,5 +309,5 @@ Keyboard Shortcuts
diff --git a/res_integration/index.html b/res_integration/index.html
index 63ee3c43..bc5319be 100644
--- a/res_integration/index.html
+++ b/res_integration/index.html
@@ -146,9 +146,14 @@ RES Integration
subnet-xxxxx |
-| SubmitterSecurityGroupIds |
-The security group names and ids used by RES VDIs. The name will be something like EnvironmentName-vdc-dcv-host-security-group |
-EnvironmentName-VDISG: sg-xxxxxxxx |
+slurm/SlurmCtl/AdditionalSecurityGroups |
+Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. |
+ |
+
+
+| slurm/InstanceConfig/AdditionalSecurityGroups |
+Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. |
+ |
| SubmitterInstanceTags |
diff --git a/search/search_index.json b/search/search_index.json
index 5f828980..c29d7720 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"AWS EDA Slurm Cluster This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters. Operating System and Processor Architecture Support This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture. Documentation View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs Security See CONTRIBUTING for more information. License This library is licensed under the MIT-0 License. See the LICENSE file.","title":"AWS EDA Slurm Cluster"},{"location":"#aws-eda-slurm-cluster","text":"This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters.","title":"AWS EDA Slurm Cluster"},{"location":"#operating-system-and-processor-architecture-support","text":"This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture.","title":"Operating System and Processor Architecture Support"},{"location":"#documentation","text":"View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs","title":"Documentation"},{"location":"#security","text":"See CONTRIBUTING for more information.","title":"Security"},{"location":"#license","text":"This library is licensed under the MIT-0 License. See the LICENSE file.","title":"License"},{"location":"CONTRIBUTING/","text":"Contributing Guidelines Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. Reporting Bugs/Feature Requests We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request . Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. Code of Conduct This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments. Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue. Licensing See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#contributing-guidelines","text":"Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#reporting-bugsfeature-requests","text":"We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment","title":"Reporting Bugs/Feature Requests"},{"location":"CONTRIBUTING/#contributing-via-pull-requests","text":"Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request .","title":"Contributing via Pull Requests"},{"location":"CONTRIBUTING/#finding-contributions-to-work-on","text":"Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.","title":"Finding contributions to work on"},{"location":"CONTRIBUTING/#code-of-conduct","text":"This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments.","title":"Code of Conduct"},{"location":"CONTRIBUTING/#security-issue-notifications","text":"If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue.","title":"Security issue notifications"},{"location":"CONTRIBUTING/#licensing","text":"See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Licensing"},{"location":"config/","text":"Configuraton File Format This project creates a ParallelCluster configuration file that is documented in the ParallelCluster User Guide . termination_protection : bool StackName : str Region : str SshKeyPair : str VpcId : str CIDR : str SubnetId : str ErrorSnsTopicArn : str TimeZone : str RESEnvironmentName : str slurm : ParallelClusterConfig : Version : str ClusterConfig : dict Image : Os : str CustomAmi : str Architecture : str ComputeNodeAmi : str DisableSimultaneousMultithreading : str EnableEfa : bool Database : DatabaseStackName : str FQDN : str Port : str AdminUserName : str AdminPasswordSecretArn : str ClientSecurityGroup : SecurityGroupName: SecurityGroupId Dcv: Enabled : bool Port : int AllowedIps : str LoginNodes : Pools : - Name : str Count : int InstanceType : str GracetimePeriod : int Image : CustomAmi : str Ssh : KeyName : str Networking : SubnetIds : - str SecurityGroups : - str AdditionalSecurityGroups : - str Iam : InstanceRole : str InstanceProfile : str AdditionalIamPolicies : - Policy : str ClusterName : str MungeKeySecret : str SlurmCtl : SlurmdPort : int instance_type : str volume_size : str CloudWatchPeriod : int PreemptMode : str PreemptType : str PreemptExemptTime : str SlurmConfOverrides : str SlurmrestdUid : str AdditionalSecurityGroups : - str AdditionalIamPolicies : - str Imds : Secured : bool SubmitterInstanceTags : str TagName: - TagValues InstanceConfig : UseSpot : str Exclude : InstanceFamilies : - str InstanceTypes : - str Include : MaxSizeOnly : bool InstanceFamilies : - str InstanceTypes : - str NodeCounts : DefaultMinCount : str DefaultMaxCount : str ComputeResourceCounts : str: # ComputeResourceName MinCount : int MaxCount : int AdditionalSecurityGroups : - str AdditionalIamPolicies : - str OnPremComputeNodes : ConfigFile : str CIDR : str Partition : str SlurmUid : int storage : ExtraMounts : - dest : str src : str type : str options : str StorageType : str FileSystemId : str VolumeId : str ExtraMountSecurityGroups : FileSystemType : SecurityGroupName: SecurityGroupId Licenses : LicenseName : Count : int Server : str Port : str ServerType : StatusScript : Top Level Config termination_protection Enable Cloudformation Stack termination protection default=True StackName The name of the configuration stack that will configure ParallelCluster and deploy it. If you do not specify the ClusterName then it will default to a value based on the StackName. If StackName ends in -config then ClusterName will be the StackName with -config stripped off. Otherwise it will be the StackName with -cl (for cluster) appended. Optional so can be specified on the command-line default='slurm-config' Region AWS region where the cluster will be deployed. Optional so can be specified on the command-line SshKeyPair Default EC2 key pair that will be used for all cluster instances. Optional so can be specified on the command-line VpcId The ID of the VPC where the cluster will be deployed. Optional so can be specified on the command-line CIDR The CIDR of the VPC. This is used in security group rules. SubnetId The ID of the VPC subnet where the cluster will be deployed. Optional. If not specified then the first private subnet is chosen. If no private subnets exist, then the first isolated subnet is chosen. If no isolated subnets exist, the the first public subnet is chosen. We recommend using a private or isolated subnet. ErrorSnsTopicArn The ARN of an existing SNS topic. Errors will be published to the SNS topic. You can subscribe to the topic so that you are notified for things like script or lambda errors. Optional, but highly recommended TimeZone The time zone to use for all EC2 instances in the cluster. default='US/Central' RESEnvironmentName If you are deploying the cluster to use from Research and Engineering Studio (RES) virtual desktops, then you can specify the environment name so that the virtual desktops automatically get configured to use the cluster. The security group of the desktops will be updated with rules that allow them to talk to the cluster and the cluster will be configured on the desktop. The Slurm binaries will be compiled for the OS of the desktops and and environment modulefile will be created so that the users just need to load the cluster modulefile to use the cluster. slurm Slurm configuration parameters. ParallelClusterConfig ParallelCluster specific configuration parameters. Version The ParallelCluster version. This is required and cannot be changed after the cluster is created. Updating to a new version of ParallelCluster requires either deleting the current cluster or creating a new cluster. ClusterConfig type: dict Additional ParallelCluster configuration settings that will be directly added to the configuration without checking. This will will be used to create the initial ParallelCluster configuration and other settings in this configuration file will override values in the dict. This exists to enable further customization of ParallelCluster beyond what this configuration supports. Image The OS and AMI to use for the head node and compute nodes. OS See the ParallelCluster docs for the supported OS distributions and versions. CustomAmi See the ParallelCluster docs for the custom AMI documentation. NOTE : A CustomAmi must be provided for Rocky8 or Rocky9. All other distributions have a default AMI that is provided by ParallelCluster. Architecture The CPU architecture to use for the cluster. ParallelCluster doesn't support heterogeneous clusters. All of the instances must have the same CPU architecture and the same OS. The cluster, however, can be accessed from login nodes of any architecture and OS. Valid Values: arm64 x86_64 default: x86_64 ComputeNodeAmi AMI to use for compute nodes. All compute nodes will use the same AMI. The default AMI is selected by the Image parameters. DisableSimultaneousMultithreading type: bool default=True Disable SMT on the compute nodes. If true, multithreading on the compute nodes is disabled. Not all instance types can disable multithreading. For a list of instance types that support disabling multithreading, see CPU cores and threads for each CPU core per instance type in the Amazon EC2 User Guide for Linux Instances. Update policy: The compute fleet must be stopped for this setting to be changed for an update. ParallelCluster documentation EnableEfa type: bool default: False Recommend to not use EFA unless necessary to avoid insufficient capacity errors when starting new instances in group or when multiple instance types in the group. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-cluster Database Optional Configure the Slurm database to use with the cluster. This is created independently of the cluster so that the same database can be used with multiple clusters. The easiest way to do this is to use the CloudFormation template provided by ParallelCluster and then to just pass the name of the stack in DatabaseStackName . All of the other parameters will be pulled from the stack. See the ParallelCluster documentation . DatabaseStackName Name of the ParallelCluster CloudFormation stack that created the database. The following parameters will be set using the outputs of the stack: FQDN Port AdminUserName AdminPasswordSecretArn ClientSecurityGroup FQDN Used with the Port to set the Uri of the database. Port type: int Database's port. AdminUserName type: str The identity that Slurm uses to connect to the database, write accounting logs, and perform queries. The user must have both read and write permissions on the database. Sets the UserName parameter in ParallelCluster. AdminPasswordSecretArn type: str The Amazon Resource Name (ARN) of the AWS Secrets Manager secret that contains the AdminUserName plaintext password. This password is used together with AdminUserName and Slurm accounting to authenticate on the database server. Sets the PasswordSecretArn parameter in ParallelCluster. ClientSecurityGroup Security group that has permissions to connect to the database. Required to be attached to the head node that is running slurmdbd so that the port connection to the database is allows. ClusterName Name of the ParallelCluster cluster. Default: If StackName ends with \"-config\" then ClusterName is StackName with \"-config\" stripped off. Otherwise add \"-cl\" to end of StackName. MungeKeySecret AWS secret with a base64 encoded munge key to use for the cluster. For an existing secret can be the secret name or the ARN. If the secret doesn't exist one will be created, but won't be part of the cloudformation stack so that it won't be deleted when the stack is deleted. Required if your submitters need to use more than 1 cluster. SlurmCtl Configure the Slurm head node or controller. Required, but can be an empty dict to accept all of the defaults. SlurmdPort Port used for the slurmd daemon on the compute nodes. default=6818 type: int instance_type Instance type of the head node. Must match the architecture of the cluster. volume_size The size of the EBS root volume on the head node in GB. default=200 type: int CloudWatchPeriod The frequency of CloudWatch metrics in seconds. default=5 type: int PreemptMode Set job preemption policy for the cluster. Jobs can be set to be preemptible when they are submitted. This allows higher priority jobs to preempt a running job when resources are constrained. This policy sets what happens to the preempted jobs. Slurm documentation Valid values: 'OFF' 'CANCEL' 'GANG' 'REQUEUE' 'SUSPEND' default='REQUEUE' PreemptType Slurm documentation Valid values: 'preempt/none' 'preempt/partition_prio' 'preempt/qos' default='preempt/partition_prio' PreemptExemptTime Slurm documentation Global option for minimum run time for all jobs before they can be considered for preemption. A time of -1 disables the option, equivalent to 0. Acceptable time formats include \"minutes\", \"minutes:seconds\", \"hours:minutes:seconds\", \"days-hours\", \"days-hours:minutes\", and \"days-hours:minutes:seconds\". default='0' type: str SlurmConfOverrides File that will be included at end of slurm.conf to override configuration parameters. This allows you to customize the slurm configuration arbitrarily. This should be used with caution since it can result in errors that make the cluster non-functional. type: str SlurmrestdUid User ID for the slurmrestd daemon. type: int default=901 SlurmRestApiVersion The REST API version. This is automatically set based on the Slurm version being used by the ParallelCluster version. type: str default: ''0.0.39' Head Node AdditionalSecurityGroups Additional security groups that will be added to the head node instance. Head Node AdditionalIamPolicies List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the head node instance. SubmitterInstanceTags Tags of instances that can be configured to submit to the cluster. When the cluster is deleted, the tag is used to unmount the slurm filesystem from the instances using SSM. InstanceConfig Configure the instances used by the cluster. A partition will be created for each combination of Base OS, Architecture, and Spot. UseSpot Configure spot instances. type: bool default: True Exclude Instance families and types to exclude. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end. Exclude InstanceFamilies Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_families = [ 'a1', # Graviton 1 'c4', # Replaced by c5 'd2', # SSD optimized 'g3', # Replaced by g4 'g3s', # Replaced by g4 'h1', # SSD optimized 'i3', # SSD optimized 'i3en', # SSD optimized 'm4', # Replaced by m5 'p2', # Replaced by p3 'p3', 'p3dn', 'r4', # Replaced by r5 't2', # Replaced by t3 'x1', 'x1e', ] Exclude InstanceTypes Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_types = [ '.+\\.(micro|nano)', # Not enough memory '.*\\.metal.*' ] Include Instance families and types to include. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end. MaxSizeOnly type: bool default: False If MaxSizeOnly is True then only the largest instance type in a family will be included unless specific instance types are included. Include InstanceFamilies Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_families = [ 'c7a', # AMD EPYC 9R14 Processor 3.7 GHz 'c7g', # AWS Graviton3 Processor 2.6 GHz # 'c7gd', # AWS Graviton3 Processor 2.6 GHz # 'c7gn', # AWS Graviton3 Processor 2.6 GHz # 'c7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz #'f1', # Intel Xeon E5-2686 v4 (Broadwell) 2.3 GHz 'm5zn', # Intel Xeon Platinum 8252 4.5 GHz 'm7a', # AMD EPYC 9R14 Processor 3.7 GHz # 'm7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'm7g', # AWS Graviton3 Processor 2.6 GHz # 'm7gd', # AWS Graviton3 Processor 2.6 GHz 'r7a', # AMD EPYC 9R14 Processor 3.7 GHz 'r7g', # AWS Graviton3 Processor 2.6 GHz # 'r7gd', # AWS Graviton3 Processor 2.6 GHz # 'r7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'r7iz', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'x2gd', # AWS Graviton2 Processor 2.5 GHz 1TB 'x2idn', # Intel Xeon Scalable (Icelake) 3.5 GHz 2 TB 'x2iedn', # Intel Xeon Scalable (Icelake) 3.5 GHz 4 TB 'x2iezn', # Intel Xeon Platinum 8252 4.5 GHz 1.5 TB #'u-6tb1', # Intel Xeon Scalable (Skylake) 6 TB #'u-9tb1', # Intel Xeon Scalable (Skylake) 9 TB #'u-12tb1', # Intel Xeon Scalable (Skylake) 12 TB ] Include InstanceTypes Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_types = [ #'c5\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz #'c5d\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5d\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz ] NodeCounts Configure the number of compute nodes of each instance type. DefaultMinCount type: int default: 0 Minimum number of compute nodes to keep running in a compute resource. If the number is greater than zero then static nodes will be created. DefaultMaxCount type: int The maximum number of compute nodes to create in a compute resource. ComputeResourceCounts Define compute node counts per compute resource. These counts will override the defaults set by DefaultMinCount and DefaultMaxCount . ComputeResourceName Name of the ParallelCluster compute resource. Can be found using sinfo . # Compute Resource MinCount type: int default: 0 # Compute Resource MaxCount type: int Compute Node AdditionalSecurityGroups Additional security groups that will be added to the compute node instances. Compute Node AdditionalIamPolicies List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the compute node instances. OnPremComputeNodes Define on-premises compute nodes that will be managed by the ParallelCluster head node. The compute nodes must be accessible from the head node over the network and any firewalls must allow all of the Slurm ports between the head node and compute nodes. ParallelCluster will be configured to allow the neccessary network traffic and the on-premises firewall can be configured to match the ParallelCluster seccurity groups. ConfigFile Configuration file with the on-premises compute nodes defined in Slurm NodeName format as described in the Slurm slurm.conf documentation . The file will be included in the ParallelCluster slurm.conf so it can technically include any Slurm configuration updates including custom partition definitions. NOTE : The syntax of the file isn't checked and syntax errors can result in the slurmctld daemon failing on the head node. On-Premises CIDR The CIDR that contains the on-premises compute nodes. This is to allow egress from the head node to the on-premises nodes. Partition A partition that will contain all of the on-premises nodes. SlurmUid type: int default: 900 The user id of the slurm user. storage ExtraMounts Additional mounts for compute nodes. This can be used so the compute nodes have the same file structure as the remote desktops. This is used to configure ParallelCluster SharedStorage . dest The directory where the file system will be mounted. This sets the MountDir . src The source path on the file system export that will be mounted. type The type of mount. For example, nfs3. options Mount options. StorageType The type of file system to mount. Valid values: Efs FsxLustre FsxOntap FsxOpenZfs FileSystemId Specifies the ID of an existing FSx for Lustre or EFS file system. VolumeId Specifies the volume ID of an existing FSx for ONTAP or FSx for OpenZFS file system. ExtraMountSecurityGroups The security groups used by the file systems so that the head and comnpute nodes can be configured to connect to them. For example: storage: ExtraMounts: - dest: \"/tools\" StorageType: FsxOpenZfs VolumeId: 'fsvol-abcd1234' src: 'fs-efgh5678.fsx.us-east-1.amazonaws.com:/fsx/' type: nfs4 options: 'nfsvers=4.1' ExtraMountSecurityGroups: zfs: FsxSG: sg-12345678 FileSystemType Type of file system so that the appropriate ports can be opened. Valid values: nfs zfs lustre Licenses Configure license counts for the scheduler. If the Slurm database is configured then it will be updated with the license counts. Otherwise, the license counts will be added to slurm.conf. LicenseName The name of the license, for example, VCSCompiler_Net or VCSMXRunTime_Net . This is the license name that users specify when submitting a job. It doesn't have to match the license name reported by the license server, although that probably makes the most sense. Count The number of licenses available to Slurm to use to schedule jobs. Once all of the license are used by running jobs, then any pending jobs will remain pending until a license becomes available. Server The license server hosting the licenses. Not currently used. Port The port on the license server used to request licenses. Not currently used. ServerType The type of license server, such as FlexLM. Not currently used. StatusScript A script that queries the license server and dynamically updates the Slurm database with the actual total number of licenses and the number used. Not currently implemented.","title":"Configuraton File Format"},{"location":"config/#configuraton-file-format","text":"This project creates a ParallelCluster configuration file that is documented in the ParallelCluster User Guide . termination_protection : bool StackName : str Region : str SshKeyPair : str VpcId : str CIDR : str SubnetId : str ErrorSnsTopicArn : str TimeZone : str RESEnvironmentName : str slurm : ParallelClusterConfig : Version : str ClusterConfig : dict Image : Os : str CustomAmi : str Architecture : str ComputeNodeAmi : str DisableSimultaneousMultithreading : str EnableEfa : bool Database : DatabaseStackName : str FQDN : str Port : str AdminUserName : str AdminPasswordSecretArn : str ClientSecurityGroup : SecurityGroupName: SecurityGroupId Dcv: Enabled : bool Port : int AllowedIps : str LoginNodes : Pools : - Name : str Count : int InstanceType : str GracetimePeriod : int Image : CustomAmi : str Ssh : KeyName : str Networking : SubnetIds : - str SecurityGroups : - str AdditionalSecurityGroups : - str Iam : InstanceRole : str InstanceProfile : str AdditionalIamPolicies : - Policy : str ClusterName : str MungeKeySecret : str SlurmCtl : SlurmdPort : int instance_type : str volume_size : str CloudWatchPeriod : int PreemptMode : str PreemptType : str PreemptExemptTime : str SlurmConfOverrides : str SlurmrestdUid : str AdditionalSecurityGroups : - str AdditionalIamPolicies : - str Imds : Secured : bool SubmitterInstanceTags : str TagName: - TagValues InstanceConfig : UseSpot : str Exclude : InstanceFamilies : - str InstanceTypes : - str Include : MaxSizeOnly : bool InstanceFamilies : - str InstanceTypes : - str NodeCounts : DefaultMinCount : str DefaultMaxCount : str ComputeResourceCounts : str: # ComputeResourceName MinCount : int MaxCount : int AdditionalSecurityGroups : - str AdditionalIamPolicies : - str OnPremComputeNodes : ConfigFile : str CIDR : str Partition : str SlurmUid : int storage : ExtraMounts : - dest : str src : str type : str options : str StorageType : str FileSystemId : str VolumeId : str ExtraMountSecurityGroups : FileSystemType : SecurityGroupName: SecurityGroupId Licenses : LicenseName : Count : int Server : str Port : str ServerType : StatusScript :","title":"Configuraton File Format"},{"location":"config/#top-level-config","text":"","title":"Top Level Config"},{"location":"config/#termination_protection","text":"Enable Cloudformation Stack termination protection default=True","title":"termination_protection"},{"location":"config/#stackname","text":"The name of the configuration stack that will configure ParallelCluster and deploy it. If you do not specify the ClusterName then it will default to a value based on the StackName. If StackName ends in -config then ClusterName will be the StackName with -config stripped off. Otherwise it will be the StackName with -cl (for cluster) appended. Optional so can be specified on the command-line default='slurm-config'","title":"StackName"},{"location":"config/#region","text":"AWS region where the cluster will be deployed. Optional so can be specified on the command-line","title":"Region"},{"location":"config/#sshkeypair","text":"Default EC2 key pair that will be used for all cluster instances. Optional so can be specified on the command-line","title":"SshKeyPair"},{"location":"config/#vpcid","text":"The ID of the VPC where the cluster will be deployed. Optional so can be specified on the command-line","title":"VpcId"},{"location":"config/#cidr","text":"The CIDR of the VPC. This is used in security group rules.","title":"CIDR"},{"location":"config/#subnetid","text":"The ID of the VPC subnet where the cluster will be deployed. Optional. If not specified then the first private subnet is chosen. If no private subnets exist, then the first isolated subnet is chosen. If no isolated subnets exist, the the first public subnet is chosen. We recommend using a private or isolated subnet.","title":"SubnetId"},{"location":"config/#errorsnstopicarn","text":"The ARN of an existing SNS topic. Errors will be published to the SNS topic. You can subscribe to the topic so that you are notified for things like script or lambda errors. Optional, but highly recommended","title":"ErrorSnsTopicArn"},{"location":"config/#timezone","text":"The time zone to use for all EC2 instances in the cluster. default='US/Central'","title":"TimeZone"},{"location":"config/#resenvironmentname","text":"If you are deploying the cluster to use from Research and Engineering Studio (RES) virtual desktops, then you can specify the environment name so that the virtual desktops automatically get configured to use the cluster. The security group of the desktops will be updated with rules that allow them to talk to the cluster and the cluster will be configured on the desktop. The Slurm binaries will be compiled for the OS of the desktops and and environment modulefile will be created so that the users just need to load the cluster modulefile to use the cluster.","title":"RESEnvironmentName"},{"location":"config/#slurm","text":"Slurm configuration parameters.","title":"slurm"},{"location":"config/#parallelclusterconfig","text":"ParallelCluster specific configuration parameters.","title":"ParallelClusterConfig"},{"location":"config/#version","text":"The ParallelCluster version. This is required and cannot be changed after the cluster is created. Updating to a new version of ParallelCluster requires either deleting the current cluster or creating a new cluster.","title":"Version"},{"location":"config/#clusterconfig","text":"type: dict Additional ParallelCluster configuration settings that will be directly added to the configuration without checking. This will will be used to create the initial ParallelCluster configuration and other settings in this configuration file will override values in the dict. This exists to enable further customization of ParallelCluster beyond what this configuration supports.","title":"ClusterConfig"},{"location":"config/#image","text":"The OS and AMI to use for the head node and compute nodes.","title":"Image"},{"location":"config/#os","text":"See the ParallelCluster docs for the supported OS distributions and versions.","title":"OS"},{"location":"config/#customami","text":"See the ParallelCluster docs for the custom AMI documentation. NOTE : A CustomAmi must be provided for Rocky8 or Rocky9. All other distributions have a default AMI that is provided by ParallelCluster.","title":"CustomAmi"},{"location":"config/#architecture","text":"The CPU architecture to use for the cluster. ParallelCluster doesn't support heterogeneous clusters. All of the instances must have the same CPU architecture and the same OS. The cluster, however, can be accessed from login nodes of any architecture and OS. Valid Values: arm64 x86_64 default: x86_64","title":"Architecture"},{"location":"config/#computenodeami","text":"AMI to use for compute nodes. All compute nodes will use the same AMI. The default AMI is selected by the Image parameters.","title":"ComputeNodeAmi"},{"location":"config/#disablesimultaneousmultithreading","text":"type: bool default=True Disable SMT on the compute nodes. If true, multithreading on the compute nodes is disabled. Not all instance types can disable multithreading. For a list of instance types that support disabling multithreading, see CPU cores and threads for each CPU core per instance type in the Amazon EC2 User Guide for Linux Instances. Update policy: The compute fleet must be stopped for this setting to be changed for an update. ParallelCluster documentation","title":"DisableSimultaneousMultithreading"},{"location":"config/#enableefa","text":"type: bool default: False Recommend to not use EFA unless necessary to avoid insufficient capacity errors when starting new instances in group or when multiple instance types in the group. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-cluster","title":"EnableEfa"},{"location":"config/#database","text":"Optional Configure the Slurm database to use with the cluster. This is created independently of the cluster so that the same database can be used with multiple clusters. The easiest way to do this is to use the CloudFormation template provided by ParallelCluster and then to just pass the name of the stack in DatabaseStackName . All of the other parameters will be pulled from the stack. See the ParallelCluster documentation .","title":"Database"},{"location":"config/#databasestackname","text":"Name of the ParallelCluster CloudFormation stack that created the database. The following parameters will be set using the outputs of the stack: FQDN Port AdminUserName AdminPasswordSecretArn ClientSecurityGroup","title":"DatabaseStackName"},{"location":"config/#fqdn","text":"Used with the Port to set the Uri of the database.","title":"FQDN"},{"location":"config/#port","text":"type: int Database's port.","title":"Port"},{"location":"config/#adminusername","text":"type: str The identity that Slurm uses to connect to the database, write accounting logs, and perform queries. The user must have both read and write permissions on the database. Sets the UserName parameter in ParallelCluster.","title":"AdminUserName"},{"location":"config/#adminpasswordsecretarn","text":"type: str The Amazon Resource Name (ARN) of the AWS Secrets Manager secret that contains the AdminUserName plaintext password. This password is used together with AdminUserName and Slurm accounting to authenticate on the database server. Sets the PasswordSecretArn parameter in ParallelCluster.","title":"AdminPasswordSecretArn"},{"location":"config/#clientsecuritygroup","text":"Security group that has permissions to connect to the database. Required to be attached to the head node that is running slurmdbd so that the port connection to the database is allows.","title":"ClientSecurityGroup"},{"location":"config/#clustername","text":"Name of the ParallelCluster cluster. Default: If StackName ends with \"-config\" then ClusterName is StackName with \"-config\" stripped off. Otherwise add \"-cl\" to end of StackName.","title":"ClusterName"},{"location":"config/#mungekeysecret","text":"AWS secret with a base64 encoded munge key to use for the cluster. For an existing secret can be the secret name or the ARN. If the secret doesn't exist one will be created, but won't be part of the cloudformation stack so that it won't be deleted when the stack is deleted. Required if your submitters need to use more than 1 cluster.","title":"MungeKeySecret"},{"location":"config/#slurmctl","text":"Configure the Slurm head node or controller. Required, but can be an empty dict to accept all of the defaults.","title":"SlurmCtl"},{"location":"config/#slurmdport","text":"Port used for the slurmd daemon on the compute nodes. default=6818 type: int","title":"SlurmdPort"},{"location":"config/#instance_type","text":"Instance type of the head node. Must match the architecture of the cluster.","title":"instance_type"},{"location":"config/#volume_size","text":"The size of the EBS root volume on the head node in GB. default=200 type: int","title":"volume_size"},{"location":"config/#cloudwatchperiod","text":"The frequency of CloudWatch metrics in seconds. default=5 type: int","title":"CloudWatchPeriod"},{"location":"config/#preemptmode","text":"Set job preemption policy for the cluster. Jobs can be set to be preemptible when they are submitted. This allows higher priority jobs to preempt a running job when resources are constrained. This policy sets what happens to the preempted jobs. Slurm documentation Valid values: 'OFF' 'CANCEL' 'GANG' 'REQUEUE' 'SUSPEND' default='REQUEUE'","title":"PreemptMode"},{"location":"config/#preempttype","text":"Slurm documentation Valid values: 'preempt/none' 'preempt/partition_prio' 'preempt/qos' default='preempt/partition_prio'","title":"PreemptType"},{"location":"config/#preemptexempttime","text":"Slurm documentation Global option for minimum run time for all jobs before they can be considered for preemption. A time of -1 disables the option, equivalent to 0. Acceptable time formats include \"minutes\", \"minutes:seconds\", \"hours:minutes:seconds\", \"days-hours\", \"days-hours:minutes\", and \"days-hours:minutes:seconds\". default='0' type: str","title":"PreemptExemptTime"},{"location":"config/#slurmconfoverrides","text":"File that will be included at end of slurm.conf to override configuration parameters. This allows you to customize the slurm configuration arbitrarily. This should be used with caution since it can result in errors that make the cluster non-functional. type: str","title":"SlurmConfOverrides"},{"location":"config/#slurmrestduid","text":"User ID for the slurmrestd daemon. type: int default=901","title":"SlurmrestdUid"},{"location":"config/#slurmrestapiversion","text":"The REST API version. This is automatically set based on the Slurm version being used by the ParallelCluster version. type: str default: ''0.0.39'","title":"SlurmRestApiVersion"},{"location":"config/#head-node-additionalsecuritygroups","text":"Additional security groups that will be added to the head node instance.","title":"Head Node AdditionalSecurityGroups"},{"location":"config/#head-node-additionaliampolicies","text":"List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the head node instance.","title":"Head Node AdditionalIamPolicies"},{"location":"config/#submitterinstancetags","text":"Tags of instances that can be configured to submit to the cluster. When the cluster is deleted, the tag is used to unmount the slurm filesystem from the instances using SSM.","title":"SubmitterInstanceTags"},{"location":"config/#instanceconfig","text":"Configure the instances used by the cluster. A partition will be created for each combination of Base OS, Architecture, and Spot.","title":"InstanceConfig"},{"location":"config/#usespot","text":"Configure spot instances. type: bool default: True","title":"UseSpot"},{"location":"config/#exclude","text":"Instance families and types to exclude. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end.","title":"Exclude"},{"location":"config/#exclude-instancefamilies","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_families = [ 'a1', # Graviton 1 'c4', # Replaced by c5 'd2', # SSD optimized 'g3', # Replaced by g4 'g3s', # Replaced by g4 'h1', # SSD optimized 'i3', # SSD optimized 'i3en', # SSD optimized 'm4', # Replaced by m5 'p2', # Replaced by p3 'p3', 'p3dn', 'r4', # Replaced by r5 't2', # Replaced by t3 'x1', 'x1e', ]","title":"Exclude InstanceFamilies"},{"location":"config/#exclude-instancetypes","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_types = [ '.+\\.(micro|nano)', # Not enough memory '.*\\.metal.*' ]","title":"Exclude InstanceTypes"},{"location":"config/#include","text":"Instance families and types to include. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end.","title":"Include"},{"location":"config/#maxsizeonly","text":"type: bool default: False If MaxSizeOnly is True then only the largest instance type in a family will be included unless specific instance types are included.","title":"MaxSizeOnly"},{"location":"config/#include-instancefamilies","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_families = [ 'c7a', # AMD EPYC 9R14 Processor 3.7 GHz 'c7g', # AWS Graviton3 Processor 2.6 GHz # 'c7gd', # AWS Graviton3 Processor 2.6 GHz # 'c7gn', # AWS Graviton3 Processor 2.6 GHz # 'c7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz #'f1', # Intel Xeon E5-2686 v4 (Broadwell) 2.3 GHz 'm5zn', # Intel Xeon Platinum 8252 4.5 GHz 'm7a', # AMD EPYC 9R14 Processor 3.7 GHz # 'm7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'm7g', # AWS Graviton3 Processor 2.6 GHz # 'm7gd', # AWS Graviton3 Processor 2.6 GHz 'r7a', # AMD EPYC 9R14 Processor 3.7 GHz 'r7g', # AWS Graviton3 Processor 2.6 GHz # 'r7gd', # AWS Graviton3 Processor 2.6 GHz # 'r7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'r7iz', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'x2gd', # AWS Graviton2 Processor 2.5 GHz 1TB 'x2idn', # Intel Xeon Scalable (Icelake) 3.5 GHz 2 TB 'x2iedn', # Intel Xeon Scalable (Icelake) 3.5 GHz 4 TB 'x2iezn', # Intel Xeon Platinum 8252 4.5 GHz 1.5 TB #'u-6tb1', # Intel Xeon Scalable (Skylake) 6 TB #'u-9tb1', # Intel Xeon Scalable (Skylake) 9 TB #'u-12tb1', # Intel Xeon Scalable (Skylake) 12 TB ]","title":"Include InstanceFamilies"},{"location":"config/#include-instancetypes","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_types = [ #'c5\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz #'c5d\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5d\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz ]","title":"Include InstanceTypes"},{"location":"config/#nodecounts","text":"Configure the number of compute nodes of each instance type.","title":"NodeCounts"},{"location":"config/#defaultmincount","text":"type: int default: 0 Minimum number of compute nodes to keep running in a compute resource. If the number is greater than zero then static nodes will be created.","title":"DefaultMinCount"},{"location":"config/#defaultmaxcount","text":"type: int The maximum number of compute nodes to create in a compute resource.","title":"DefaultMaxCount"},{"location":"config/#computeresourcecounts","text":"Define compute node counts per compute resource. These counts will override the defaults set by DefaultMinCount and DefaultMaxCount .","title":"ComputeResourceCounts"},{"location":"config/#computeresourcename","text":"Name of the ParallelCluster compute resource. Can be found using sinfo .","title":"ComputeResourceName"},{"location":"config/#compute-resource-mincount","text":"type: int default: 0","title":"# Compute Resource MinCount"},{"location":"config/#compute-resource-maxcount","text":"type: int","title":"# Compute Resource MaxCount"},{"location":"config/#compute-node-additionalsecuritygroups","text":"Additional security groups that will be added to the compute node instances.","title":"Compute Node AdditionalSecurityGroups"},{"location":"config/#compute-node-additionaliampolicies","text":"List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the compute node instances.","title":"Compute Node AdditionalIamPolicies"},{"location":"config/#onpremcomputenodes","text":"Define on-premises compute nodes that will be managed by the ParallelCluster head node. The compute nodes must be accessible from the head node over the network and any firewalls must allow all of the Slurm ports between the head node and compute nodes. ParallelCluster will be configured to allow the neccessary network traffic and the on-premises firewall can be configured to match the ParallelCluster seccurity groups.","title":"OnPremComputeNodes"},{"location":"config/#configfile","text":"Configuration file with the on-premises compute nodes defined in Slurm NodeName format as described in the Slurm slurm.conf documentation . The file will be included in the ParallelCluster slurm.conf so it can technically include any Slurm configuration updates including custom partition definitions. NOTE : The syntax of the file isn't checked and syntax errors can result in the slurmctld daemon failing on the head node.","title":"ConfigFile"},{"location":"config/#on-premises-cidr","text":"The CIDR that contains the on-premises compute nodes. This is to allow egress from the head node to the on-premises nodes.","title":"On-Premises CIDR"},{"location":"config/#partition","text":"A partition that will contain all of the on-premises nodes.","title":"Partition"},{"location":"config/#slurmuid","text":"type: int default: 900 The user id of the slurm user.","title":"SlurmUid"},{"location":"config/#storage","text":"","title":"storage"},{"location":"config/#extramounts","text":"Additional mounts for compute nodes. This can be used so the compute nodes have the same file structure as the remote desktops. This is used to configure ParallelCluster SharedStorage .","title":"ExtraMounts"},{"location":"config/#dest","text":"The directory where the file system will be mounted. This sets the MountDir .","title":"dest"},{"location":"config/#src","text":"The source path on the file system export that will be mounted.","title":"src"},{"location":"config/#type","text":"The type of mount. For example, nfs3.","title":"type"},{"location":"config/#options","text":"Mount options.","title":"options"},{"location":"config/#storagetype","text":"The type of file system to mount. Valid values: Efs FsxLustre FsxOntap FsxOpenZfs","title":"StorageType"},{"location":"config/#filesystemid","text":"Specifies the ID of an existing FSx for Lustre or EFS file system.","title":"FileSystemId"},{"location":"config/#volumeid","text":"Specifies the volume ID of an existing FSx for ONTAP or FSx for OpenZFS file system.","title":"VolumeId"},{"location":"config/#extramountsecuritygroups","text":"The security groups used by the file systems so that the head and comnpute nodes can be configured to connect to them. For example: storage: ExtraMounts: - dest: \"/tools\" StorageType: FsxOpenZfs VolumeId: 'fsvol-abcd1234' src: 'fs-efgh5678.fsx.us-east-1.amazonaws.com:/fsx/' type: nfs4 options: 'nfsvers=4.1' ExtraMountSecurityGroups: zfs: FsxSG: sg-12345678","title":"ExtraMountSecurityGroups"},{"location":"config/#filesystemtype","text":"Type of file system so that the appropriate ports can be opened. Valid values: nfs zfs lustre","title":"FileSystemType"},{"location":"config/#licenses","text":"Configure license counts for the scheduler. If the Slurm database is configured then it will be updated with the license counts. Otherwise, the license counts will be added to slurm.conf.","title":"Licenses"},{"location":"config/#licensename","text":"The name of the license, for example, VCSCompiler_Net or VCSMXRunTime_Net . This is the license name that users specify when submitting a job. It doesn't have to match the license name reported by the license server, although that probably makes the most sense.","title":"LicenseName"},{"location":"config/#count","text":"The number of licenses available to Slurm to use to schedule jobs. Once all of the license are used by running jobs, then any pending jobs will remain pending until a license becomes available.","title":"Count"},{"location":"config/#server","text":"The license server hosting the licenses. Not currently used.","title":"Server"},{"location":"config/#port_1","text":"The port on the license server used to request licenses. Not currently used.","title":"Port"},{"location":"config/#servertype","text":"The type of license server, such as FlexLM. Not currently used.","title":"ServerType"},{"location":"config/#statusscript","text":"A script that queries the license server and dynamically updates the Slurm database with the actual total number of licenses and the number used. Not currently implemented.","title":"StatusScript"},{"location":"custom-amis/","text":"Custom AMIs for ParallelCluster ParallelCluster supports building custom ParallelCluster AMIs for the head and compute nodes . You can specify a custom AMI for the entire cluster (head and compute nodes) and you can also specify a custom AMI for just the compute nodes. By default, ParallelCluster will use pre-built AMIs for the OS that you select. The exception is Rocky 8 and 9, for which ParallelCluster does not provide pre-built AMIs. To use Rocky Linux, you must first build a custom AMI and specify it in your config file at slurm/ParallelClusterConfig/Os/CustomAmi . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The build files can be modified for your needs. The build file format is docummented in the ParallelCluster User Guide . For example, you can add your own scripts to run during the AMI build process. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete. FPGA Developer AMI The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 . Deploy or update the Cluster After the AMI is built, add it to the config and create or update your cluster to use the AMI. You can set the AMI for the compute and head nodes using slurm/ParallelClusterConfig/Os/CustomAmi and for the compute nodes only using slurm/ParallelClusterConfig/ComputeNodeAmi . Note : You cannot update the OS of the cluster or the AMI of the head node. If they need to change then you will need to create a new cluster. The config file will look something like the following: slurm: ParallelClusterConfig: Image: Os: rocky8 CustomAmi: ami-abc123 # Rocky linux ComputeNodeAmi: ami-def456 # Rocky linux + EDA packages and EDA file systems","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#custom-amis-for-parallelcluster","text":"ParallelCluster supports building custom ParallelCluster AMIs for the head and compute nodes . You can specify a custom AMI for the entire cluster (head and compute nodes) and you can also specify a custom AMI for just the compute nodes. By default, ParallelCluster will use pre-built AMIs for the OS that you select. The exception is Rocky 8 and 9, for which ParallelCluster does not provide pre-built AMIs. To use Rocky Linux, you must first build a custom AMI and specify it in your config file at slurm/ParallelClusterConfig/Os/CustomAmi . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The build files can be modified for your needs. The build file format is docummented in the ParallelCluster User Guide . For example, you can add your own scripts to run during the AMI build process. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete.","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#fpga-developer-ami","text":"The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 .","title":"FPGA Developer AMI"},{"location":"custom-amis/#deploy-or-update-the-cluster","text":"After the AMI is built, add it to the config and create or update your cluster to use the AMI. You can set the AMI for the compute and head nodes using slurm/ParallelClusterConfig/Os/CustomAmi and for the compute nodes only using slurm/ParallelClusterConfig/ComputeNodeAmi . Note : You cannot update the OS of the cluster or the AMI of the head node. If they need to change then you will need to create a new cluster. The config file will look something like the following: slurm: ParallelClusterConfig: Image: Os: rocky8 CustomAmi: ami-abc123 # Rocky linux ComputeNodeAmi: ami-def456 # Rocky linux + EDA packages and EDA file systems","title":"Deploy or update the Cluster"},{"location":"debug/","text":"Debug For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation . Slurm Head Node If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv Compute Nodes If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd Log Files Logfile Description /var/log/slurmd.log slurmctld logfile Job Stuck in Pending State You can use scontrol to get detailed information about a job. scontrol show job *jobid* Job Stuck in Completing State When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Debug"},{"location":"debug/#debug","text":"For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation .","title":"Debug"},{"location":"debug/#slurm-head-node","text":"If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv","title":"Slurm Head Node"},{"location":"debug/#compute-nodes","text":"If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd","title":"Compute Nodes"},{"location":"debug/#log-files","text":"Logfile Description /var/log/slurmd.log slurmctld logfile","title":"Log Files"},{"location":"debug/#job-stuck-in-pending-state","text":"You can use scontrol to get detailed information about a job. scontrol show job *jobid*","title":"Job Stuck in Pending State"},{"location":"debug/#job-stuck-in-completing-state","text":"When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Job Stuck in Completing State"},{"location":"delete-cluster/","text":"Delete Cluster To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"delete-cluster/#delete-cluster","text":"To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"deploy-parallel-cluster/","text":"Deploy AWS ParallelCluster A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0. Prerequisites See Deployment Prerequisites page. Create ParallelCluster UI (optional but recommended) It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide . Create ParallelCluster Slurm Database The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting . Create the Cluster To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster. Create users_groups.json Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys. Configure submission hosts to use the cluster ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm. Customize the compute node AMI The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again. Run Your First Job Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash Slurm Documentation https://slurm.schedmd.com","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#deploy-aws-parallelcluster","text":"A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0.","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#prerequisites","text":"See Deployment Prerequisites page.","title":"Prerequisites"},{"location":"deploy-parallel-cluster/#create-parallelcluster-ui-optional-but-recommended","text":"It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide .","title":"Create ParallelCluster UI (optional but recommended)"},{"location":"deploy-parallel-cluster/#create-parallelcluster-slurm-database","text":"The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting .","title":"Create ParallelCluster Slurm Database"},{"location":"deploy-parallel-cluster/#create-the-cluster","text":"To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster.","title":"Create the Cluster"},{"location":"deploy-parallel-cluster/#create-users_groupsjson","text":"Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys.","title":"Create users_groups.json"},{"location":"deploy-parallel-cluster/#configure-submission-hosts-to-use-the-cluster","text":"ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm.","title":"Configure submission hosts to use the cluster"},{"location":"deploy-parallel-cluster/#customize-the-compute-node-ami","text":"The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again.","title":"Customize the compute node AMI"},{"location":"deploy-parallel-cluster/#run-your-first-job","text":"Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash","title":"Run Your First Job"},{"location":"deploy-parallel-cluster/#slurm-documentation","text":"https://slurm.schedmd.com","title":"Slurm Documentation"},{"location":"deployment-prerequisites/","text":"Deployment Prerequisites This page shows common prerequisites that need to be done before deployment. Deployment Server/Instance Requirements The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance. Configure AWS CLI Credentials You will needs AWS credentials that provide admin access to deploy the cluster. Clone or Download the Repository Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git Create SNS Topic for Error Notifications (Optional but recommended) The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket. Make sure using at least python version 3.7 This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5 Make sure required packages are installed cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment. Install Cloud Development Kit (CDK) (Optional) The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed. Create Configuration File Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml Configure the Compute Instances The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 RedHat 9 x86_64, arm64 Rocky 8 x86_64, arm64 Rocky 9 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1 Configure Fair Share Scheduling (Optional) Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities. Configure Licenses Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-prerequisites","text":"This page shows common prerequisites that need to be done before deployment.","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-serverinstance-requirements","text":"The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance.","title":"Deployment Server/Instance Requirements"},{"location":"deployment-prerequisites/#configure-aws-cli-credentials","text":"You will needs AWS credentials that provide admin access to deploy the cluster.","title":"Configure AWS CLI Credentials"},{"location":"deployment-prerequisites/#clone-or-download-the-repository","text":"Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git","title":"Clone or Download the Repository"},{"location":"deployment-prerequisites/#create-sns-topic-for-error-notifications-optional-but-recommended","text":"The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket.","title":"Create SNS Topic for Error Notifications (Optional but recommended)"},{"location":"deployment-prerequisites/#make-sure-using-at-least-python-version-37","text":"This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5","title":"Make sure using at least python version 3.7"},{"location":"deployment-prerequisites/#make-sure-required-packages-are-installed","text":"cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment.","title":"Make sure required packages are installed"},{"location":"deployment-prerequisites/#install-cloud-development-kit-cdk-optional","text":"The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed.","title":"Install Cloud Development Kit (CDK) (Optional)"},{"location":"deployment-prerequisites/#create-configuration-file","text":"Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml","title":"Create Configuration File"},{"location":"deployment-prerequisites/#configure-the-compute-instances","text":"The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 RedHat 9 x86_64, arm64 Rocky 8 x86_64, arm64 Rocky 9 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1","title":"Configure the Compute Instances"},{"location":"deployment-prerequisites/#configure-fair-share-scheduling-optional","text":"Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities.","title":"Configure Fair Share Scheduling (Optional)"},{"location":"deployment-prerequisites/#configure-licenses","text":"Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Configure Licenses"},{"location":"federation/","text":"Federation (legacy) To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"federation/#federation-legacy","text":"To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"implementation/","text":"Implementation Details (legacy) Slurm Infrastructure All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons. Directory Structure All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}} CloudWatch Metrics CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py Down Node Handling If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer Insufficient Capacity Exception (ICE) Handling When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Implementation Details (legacy)"},{"location":"implementation/#implementation-details-legacy","text":"","title":"Implementation Details (legacy)"},{"location":"implementation/#slurm-infrastructure","text":"All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons.","title":"Slurm Infrastructure"},{"location":"implementation/#directory-structure","text":"All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}}","title":"Directory Structure"},{"location":"implementation/#cloudwatch-metrics","text":"CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py","title":"CloudWatch Metrics"},{"location":"implementation/#down-node-handling","text":"If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer","title":"Down Node Handling"},{"location":"implementation/#insufficient-capacity-exception-ice-handling","text":"When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Insufficient Capacity Exception (ICE) Handling"},{"location":"job_preemption/","text":"Job Preemption The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license. Documentation https://slurm.schedmd.com/preempt.html","title":"Job Preemption"},{"location":"job_preemption/#job-preemption","text":"The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license.","title":"Job Preemption"},{"location":"job_preemption/#documentation","text":"https://slurm.schedmd.com/preempt.html","title":"Documentation"},{"location":"onprem/","text":"On-Premises Integration The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information. Network Requirements The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes. DNS Requirements Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC. File System Requirements All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm. Slurm Configuration of On-Premises Compute Nodes The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem Simulating an On-Premises Network Using AWS Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"On-Premises Integration"},{"location":"onprem/#on-premises-integration","text":"The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information.","title":"On-Premises Integration"},{"location":"onprem/#network-requirements","text":"The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes.","title":"Network Requirements"},{"location":"onprem/#dns-requirements","text":"Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC.","title":"DNS Requirements"},{"location":"onprem/#file-system-requirements","text":"All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm.","title":"File System Requirements"},{"location":"onprem/#slurm-configuration-of-on-premises-compute-nodes","text":"The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem","title":"Slurm Configuration of On-Premises Compute Nodes"},{"location":"onprem/#simulating-an-on-premises-network-using-aws","text":"Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"Simulating an On-Premises Network Using AWS"},{"location":"res_integration/","text":"RES Integration Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"res_integration/#res-integration","text":"Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"rest_api/","text":"Slurm REST API The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster. How to use the REST API The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"Slurm REST API"},{"location":"rest_api/#slurm-rest-api","text":"The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster.","title":"Slurm REST API"},{"location":"rest_api/#how-to-use-the-rest-api","text":"The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"How to use the REST API"},{"location":"run_jobs/","text":"Run Jobs This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages. Set Up Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format. Key Slurm Commands The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state sbatch The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script. Run a simulation build followed by a regression build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi srun The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs squeue The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue sprio Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11 sacct Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct sreport The sreport command can be used to generate report from the Slurm database. Other Slurm Commands Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Run Jobs"},{"location":"run_jobs/#run-jobs","text":"This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages.","title":"Run Jobs"},{"location":"run_jobs/#set-up","text":"Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format.","title":"Set Up"},{"location":"run_jobs/#key-slurm-commands","text":"The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state","title":"Key Slurm Commands"},{"location":"run_jobs/#sbatch","text":"The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script.","title":"sbatch"},{"location":"run_jobs/#run-a-simulation-build-followed-by-a-regression","text":"build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi","title":"Run a simulation build followed by a regression"},{"location":"run_jobs/#srun","text":"The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs","title":"srun"},{"location":"run_jobs/#squeue","text":"The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue","title":"squeue"},{"location":"run_jobs/#sprio","text":"Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11","title":"sprio"},{"location":"run_jobs/#sacct","text":"Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct","title":"sacct"},{"location":"run_jobs/#sreport","text":"The sreport command can be used to generate report from the Slurm database.","title":"sreport"},{"location":"run_jobs/#other-slurm-commands","text":"Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Other Slurm Commands"},{"location":"soca_integration/","text":"SOCA Integration Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"},{"location":"soca_integration/#soca-integration","text":"Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"AWS EDA Slurm Cluster This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters. Operating System and Processor Architecture Support This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture. Documentation View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs Security See CONTRIBUTING for more information. License This library is licensed under the MIT-0 License. See the LICENSE file.","title":"AWS EDA Slurm Cluster"},{"location":"#aws-eda-slurm-cluster","text":"This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters.","title":"AWS EDA Slurm Cluster"},{"location":"#operating-system-and-processor-architecture-support","text":"This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture.","title":"Operating System and Processor Architecture Support"},{"location":"#documentation","text":"View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs","title":"Documentation"},{"location":"#security","text":"See CONTRIBUTING for more information.","title":"Security"},{"location":"#license","text":"This library is licensed under the MIT-0 License. See the LICENSE file.","title":"License"},{"location":"CONTRIBUTING/","text":"Contributing Guidelines Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. Reporting Bugs/Feature Requests We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request . Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. Code of Conduct This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments. Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue. Licensing See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#contributing-guidelines","text":"Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#reporting-bugsfeature-requests","text":"We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment","title":"Reporting Bugs/Feature Requests"},{"location":"CONTRIBUTING/#contributing-via-pull-requests","text":"Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request .","title":"Contributing via Pull Requests"},{"location":"CONTRIBUTING/#finding-contributions-to-work-on","text":"Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.","title":"Finding contributions to work on"},{"location":"CONTRIBUTING/#code-of-conduct","text":"This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments.","title":"Code of Conduct"},{"location":"CONTRIBUTING/#security-issue-notifications","text":"If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue.","title":"Security issue notifications"},{"location":"CONTRIBUTING/#licensing","text":"See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Licensing"},{"location":"config/","text":"Configuraton File Format This project creates a ParallelCluster configuration file that is documented in the ParallelCluster User Guide . termination_protection : bool StackName : str Region : str SshKeyPair : str VpcId : str CIDR : str SubnetId : str ErrorSnsTopicArn : str TimeZone : str RESEnvironmentName : str slurm : ParallelClusterConfig : Version : str ClusterConfig : dict Image : Os : str CustomAmi : str Architecture : str ComputeNodeAmi : str DisableSimultaneousMultithreading : str EnableEfa : bool Database : DatabaseStackName : str FQDN : str Port : str AdminUserName : str AdminPasswordSecretArn : str ClientSecurityGroup : SecurityGroupName: SecurityGroupId Dcv: Enabled : bool Port : int AllowedIps : str LoginNodes : Pools : - Name : str Count : int InstanceType : str GracetimePeriod : int Image : CustomAmi : str Ssh : KeyName : str Networking : SubnetIds : - str SecurityGroups : - str AdditionalSecurityGroups : - str Iam : InstanceRole : str InstanceProfile : str AdditionalIamPolicies : - Policy : str ClusterName : str MungeKeySecret : str SlurmCtl : SlurmdPort : int instance_type : str volume_size : str CloudWatchPeriod : int PreemptMode : str PreemptType : str PreemptExemptTime : str SlurmConfOverrides : str SlurmrestdUid : str AdditionalSecurityGroups : - str AdditionalIamPolicies : - str Imds : Secured : bool SubmitterInstanceTags : str TagName: - TagValues InstanceConfig : UseSpot : str Exclude : InstanceFamilies : - str InstanceTypes : - str Include : MaxSizeOnly : bool InstanceFamilies : - str InstanceTypes : - str NodeCounts : DefaultMinCount : str DefaultMaxCount : str ComputeResourceCounts : str: # ComputeResourceName MinCount : int MaxCount : int AdditionalSecurityGroups : - str AdditionalIamPolicies : - str OnPremComputeNodes : ConfigFile : str CIDR : str Partition : str SlurmUid : int storage : ExtraMounts : - dest : str src : str type : str options : str StorageType : str FileSystemId : str VolumeId : str ExtraMountSecurityGroups : FileSystemType : SecurityGroupName: SecurityGroupId Licenses : LicenseName : Count : int Server : str Port : str ServerType : StatusScript : Top Level Config termination_protection Enable Cloudformation Stack termination protection default=True StackName The name of the configuration stack that will configure ParallelCluster and deploy it. If you do not specify the ClusterName then it will default to a value based on the StackName. If StackName ends in -config then ClusterName will be the StackName with -config stripped off. Otherwise it will be the StackName with -cl (for cluster) appended. Optional so can be specified on the command-line default='slurm-config' Region AWS region where the cluster will be deployed. Optional so can be specified on the command-line SshKeyPair Default EC2 key pair that will be used for all cluster instances. Optional so can be specified on the command-line VpcId The ID of the VPC where the cluster will be deployed. Optional so can be specified on the command-line CIDR The CIDR of the VPC. This is used in security group rules. SubnetId The ID of the VPC subnet where the cluster will be deployed. Optional. If not specified then the first private subnet is chosen. If no private subnets exist, then the first isolated subnet is chosen. If no isolated subnets exist, the the first public subnet is chosen. We recommend using a private or isolated subnet. ErrorSnsTopicArn The ARN of an existing SNS topic. Errors will be published to the SNS topic. You can subscribe to the topic so that you are notified for things like script or lambda errors. Optional, but highly recommended TimeZone The time zone to use for all EC2 instances in the cluster. default='US/Central' RESEnvironmentName If you are deploying the cluster to use from Research and Engineering Studio (RES) virtual desktops, then you can specify the environment name so that the virtual desktops automatically get configured to use the cluster. The security group of the desktops will be updated with rules that allow them to talk to the cluster and the cluster will be configured on the desktop. The Slurm binaries will be compiled for the OS of the desktops and and environment modulefile will be created so that the users just need to load the cluster modulefile to use the cluster. slurm Slurm configuration parameters. ParallelClusterConfig ParallelCluster specific configuration parameters. Version The ParallelCluster version. This is required and cannot be changed after the cluster is created. Updating to a new version of ParallelCluster requires either deleting the current cluster or creating a new cluster. ClusterConfig type: dict Additional ParallelCluster configuration settings that will be directly added to the configuration without checking. This will will be used to create the initial ParallelCluster configuration and other settings in this configuration file will override values in the dict. This exists to enable further customization of ParallelCluster beyond what this configuration supports. Image The OS and AMI to use for the head node and compute nodes. OS See the ParallelCluster docs for the supported OS distributions and versions. CustomAmi See the ParallelCluster docs for the custom AMI documentation. NOTE : A CustomAmi must be provided for Rocky8 or Rocky9. All other distributions have a default AMI that is provided by ParallelCluster. Architecture The CPU architecture to use for the cluster. ParallelCluster doesn't support heterogeneous clusters. All of the instances must have the same CPU architecture and the same OS. The cluster, however, can be accessed from login nodes of any architecture and OS. Valid Values: arm64 x86_64 default: x86_64 ComputeNodeAmi AMI to use for compute nodes. All compute nodes will use the same AMI. The default AMI is selected by the Image parameters. DisableSimultaneousMultithreading type: bool default=True Disable SMT on the compute nodes. If true, multithreading on the compute nodes is disabled. Not all instance types can disable multithreading. For a list of instance types that support disabling multithreading, see CPU cores and threads for each CPU core per instance type in the Amazon EC2 User Guide for Linux Instances. Update policy: The compute fleet must be stopped for this setting to be changed for an update. ParallelCluster documentation EnableEfa type: bool default: False Recommend to not use EFA unless necessary to avoid insufficient capacity errors when starting new instances in group or when multiple instance types in the group. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-cluster Database Optional Configure the Slurm database to use with the cluster. This is created independently of the cluster so that the same database can be used with multiple clusters. The easiest way to do this is to use the CloudFormation template provided by ParallelCluster and then to just pass the name of the stack in DatabaseStackName . All of the other parameters will be pulled from the stack. See the ParallelCluster documentation . DatabaseStackName Name of the ParallelCluster CloudFormation stack that created the database. The following parameters will be set using the outputs of the stack: FQDN Port AdminUserName AdminPasswordSecretArn ClientSecurityGroup FQDN Used with the Port to set the Uri of the database. Port type: int Database's port. AdminUserName type: str The identity that Slurm uses to connect to the database, write accounting logs, and perform queries. The user must have both read and write permissions on the database. Sets the UserName parameter in ParallelCluster. AdminPasswordSecretArn type: str The Amazon Resource Name (ARN) of the AWS Secrets Manager secret that contains the AdminUserName plaintext password. This password is used together with AdminUserName and Slurm accounting to authenticate on the database server. Sets the PasswordSecretArn parameter in ParallelCluster. ClientSecurityGroup Security group that has permissions to connect to the database. Required to be attached to the head node that is running slurmdbd so that the port connection to the database is allows. ClusterName Name of the ParallelCluster cluster. Default: If StackName ends with \"-config\" then ClusterName is StackName with \"-config\" stripped off. Otherwise add \"-cl\" to end of StackName. MungeKeySecret AWS secret with a base64 encoded munge key to use for the cluster. For an existing secret can be the secret name or the ARN. If the secret doesn't exist one will be created, but won't be part of the cloudformation stack so that it won't be deleted when the stack is deleted. Required if your submitters need to use more than 1 cluster. SlurmCtl Configure the Slurm head node or controller. Required, but can be an empty dict to accept all of the defaults. SlurmdPort Port used for the slurmd daemon on the compute nodes. default=6818 type: int instance_type Instance type of the head node. Must match the architecture of the cluster. volume_size The size of the EBS root volume on the head node in GB. default=200 type: int CloudWatchPeriod The frequency of CloudWatch metrics in seconds. default=5 type: int PreemptMode Set job preemption policy for the cluster. Jobs can be set to be preemptible when they are submitted. This allows higher priority jobs to preempt a running job when resources are constrained. This policy sets what happens to the preempted jobs. Slurm documentation Valid values: 'OFF' 'CANCEL' 'GANG' 'REQUEUE' 'SUSPEND' default='REQUEUE' PreemptType Slurm documentation Valid values: 'preempt/none' 'preempt/partition_prio' 'preempt/qos' default='preempt/partition_prio' PreemptExemptTime Slurm documentation Global option for minimum run time for all jobs before they can be considered for preemption. A time of -1 disables the option, equivalent to 0. Acceptable time formats include \"minutes\", \"minutes:seconds\", \"hours:minutes:seconds\", \"days-hours\", \"days-hours:minutes\", and \"days-hours:minutes:seconds\". default='0' type: str SlurmConfOverrides File that will be included at end of slurm.conf to override configuration parameters. This allows you to customize the slurm configuration arbitrarily. This should be used with caution since it can result in errors that make the cluster non-functional. type: str SlurmrestdUid User ID for the slurmrestd daemon. type: int default=901 SlurmRestApiVersion The REST API version. This is automatically set based on the Slurm version being used by the ParallelCluster version. type: str default: ''0.0.39' Head Node AdditionalSecurityGroups Additional security groups that will be added to the head node instance. Head Node AdditionalIamPolicies List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the head node instance. SubmitterInstanceTags Tags of instances that can be configured to submit to the cluster. When the cluster is deleted, the tag is used to unmount the slurm filesystem from the instances using SSM. InstanceConfig Configure the instances used by the cluster. A partition will be created for each combination of Base OS, Architecture, and Spot. UseSpot Configure spot instances. type: bool default: True Exclude Instance families and types to exclude. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end. Exclude InstanceFamilies Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_families = [ 'a1', # Graviton 1 'c4', # Replaced by c5 'd2', # SSD optimized 'g3', # Replaced by g4 'g3s', # Replaced by g4 'h1', # SSD optimized 'i3', # SSD optimized 'i3en', # SSD optimized 'm4', # Replaced by m5 'p2', # Replaced by p3 'p3', 'p3dn', 'r4', # Replaced by r5 't2', # Replaced by t3 'x1', 'x1e', ] Exclude InstanceTypes Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_types = [ '.+\\.(micro|nano)', # Not enough memory '.*\\.metal.*' ] Include Instance families and types to include. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end. MaxSizeOnly type: bool default: False If MaxSizeOnly is True then only the largest instance type in a family will be included unless specific instance types are included. Include InstanceFamilies Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_families = [ 'c7a', # AMD EPYC 9R14 Processor 3.7 GHz 'c7g', # AWS Graviton3 Processor 2.6 GHz # 'c7gd', # AWS Graviton3 Processor 2.6 GHz # 'c7gn', # AWS Graviton3 Processor 2.6 GHz # 'c7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz #'f1', # Intel Xeon E5-2686 v4 (Broadwell) 2.3 GHz 'm5zn', # Intel Xeon Platinum 8252 4.5 GHz 'm7a', # AMD EPYC 9R14 Processor 3.7 GHz # 'm7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'm7g', # AWS Graviton3 Processor 2.6 GHz # 'm7gd', # AWS Graviton3 Processor 2.6 GHz 'r7a', # AMD EPYC 9R14 Processor 3.7 GHz 'r7g', # AWS Graviton3 Processor 2.6 GHz # 'r7gd', # AWS Graviton3 Processor 2.6 GHz # 'r7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'r7iz', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'x2gd', # AWS Graviton2 Processor 2.5 GHz 1TB 'x2idn', # Intel Xeon Scalable (Icelake) 3.5 GHz 2 TB 'x2iedn', # Intel Xeon Scalable (Icelake) 3.5 GHz 4 TB 'x2iezn', # Intel Xeon Platinum 8252 4.5 GHz 1.5 TB #'u-6tb1', # Intel Xeon Scalable (Skylake) 6 TB #'u-9tb1', # Intel Xeon Scalable (Skylake) 9 TB #'u-12tb1', # Intel Xeon Scalable (Skylake) 12 TB ] Include InstanceTypes Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_types = [ #'c5\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz #'c5d\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5d\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz ] NodeCounts Configure the number of compute nodes of each instance type. DefaultMinCount type: int default: 0 Minimum number of compute nodes to keep running in a compute resource. If the number is greater than zero then static nodes will be created. DefaultMaxCount type: int The maximum number of compute nodes to create in a compute resource. ComputeResourceCounts Define compute node counts per compute resource. These counts will override the defaults set by DefaultMinCount and DefaultMaxCount . ComputeResourceName Name of the ParallelCluster compute resource. Can be found using sinfo . # Compute Resource MinCount type: int default: 0 # Compute Resource MaxCount type: int Compute Node AdditionalSecurityGroups Additional security groups that will be added to the compute node instances. Compute Node AdditionalIamPolicies List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the compute node instances. OnPremComputeNodes Define on-premises compute nodes that will be managed by the ParallelCluster head node. The compute nodes must be accessible from the head node over the network and any firewalls must allow all of the Slurm ports between the head node and compute nodes. ParallelCluster will be configured to allow the neccessary network traffic and the on-premises firewall can be configured to match the ParallelCluster seccurity groups. ConfigFile Configuration file with the on-premises compute nodes defined in Slurm NodeName format as described in the Slurm slurm.conf documentation . The file will be included in the ParallelCluster slurm.conf so it can technically include any Slurm configuration updates including custom partition definitions. NOTE : The syntax of the file isn't checked and syntax errors can result in the slurmctld daemon failing on the head node. On-Premises CIDR The CIDR that contains the on-premises compute nodes. This is to allow egress from the head node to the on-premises nodes. Partition A partition that will contain all of the on-premises nodes. SlurmUid type: int default: 900 The user id of the slurm user. storage ExtraMounts Additional mounts for compute nodes. This can be used so the compute nodes have the same file structure as the remote desktops. This is used to configure ParallelCluster SharedStorage . dest The directory where the file system will be mounted. This sets the MountDir . src The source path on the file system export that will be mounted. type The type of mount. For example, nfs3. options Mount options. StorageType The type of file system to mount. Valid values: Efs FsxLustre FsxOntap FsxOpenZfs FileSystemId Specifies the ID of an existing FSx for Lustre or EFS file system. VolumeId Specifies the volume ID of an existing FSx for ONTAP or FSx for OpenZFS file system. ExtraMountSecurityGroups The security groups used by the file systems so that the head and comnpute nodes can be configured to connect to them. For example: storage: ExtraMounts: - dest: \"/tools\" StorageType: FsxOpenZfs VolumeId: 'fsvol-abcd1234' src: 'fs-efgh5678.fsx.us-east-1.amazonaws.com:/fsx/' type: nfs4 options: 'nfsvers=4.1' ExtraMountSecurityGroups: zfs: FsxSG: sg-12345678 FileSystemType Type of file system so that the appropriate ports can be opened. Valid values: nfs zfs lustre Licenses Configure license counts for the scheduler. If the Slurm database is configured then it will be updated with the license counts. Otherwise, the license counts will be added to slurm.conf. LicenseName The name of the license, for example, VCSCompiler_Net or VCSMXRunTime_Net . This is the license name that users specify when submitting a job. It doesn't have to match the license name reported by the license server, although that probably makes the most sense. Count The number of licenses available to Slurm to use to schedule jobs. Once all of the license are used by running jobs, then any pending jobs will remain pending until a license becomes available. Server The license server hosting the licenses. Not currently used. Port The port on the license server used to request licenses. Not currently used. ServerType The type of license server, such as FlexLM. Not currently used. StatusScript A script that queries the license server and dynamically updates the Slurm database with the actual total number of licenses and the number used. Not currently implemented.","title":"Configuraton File Format"},{"location":"config/#configuraton-file-format","text":"This project creates a ParallelCluster configuration file that is documented in the ParallelCluster User Guide . termination_protection : bool StackName : str Region : str SshKeyPair : str VpcId : str CIDR : str SubnetId : str ErrorSnsTopicArn : str TimeZone : str RESEnvironmentName : str slurm : ParallelClusterConfig : Version : str ClusterConfig : dict Image : Os : str CustomAmi : str Architecture : str ComputeNodeAmi : str DisableSimultaneousMultithreading : str EnableEfa : bool Database : DatabaseStackName : str FQDN : str Port : str AdminUserName : str AdminPasswordSecretArn : str ClientSecurityGroup : SecurityGroupName: SecurityGroupId Dcv: Enabled : bool Port : int AllowedIps : str LoginNodes : Pools : - Name : str Count : int InstanceType : str GracetimePeriod : int Image : CustomAmi : str Ssh : KeyName : str Networking : SubnetIds : - str SecurityGroups : - str AdditionalSecurityGroups : - str Iam : InstanceRole : str InstanceProfile : str AdditionalIamPolicies : - Policy : str ClusterName : str MungeKeySecret : str SlurmCtl : SlurmdPort : int instance_type : str volume_size : str CloudWatchPeriod : int PreemptMode : str PreemptType : str PreemptExemptTime : str SlurmConfOverrides : str SlurmrestdUid : str AdditionalSecurityGroups : - str AdditionalIamPolicies : - str Imds : Secured : bool SubmitterInstanceTags : str TagName: - TagValues InstanceConfig : UseSpot : str Exclude : InstanceFamilies : - str InstanceTypes : - str Include : MaxSizeOnly : bool InstanceFamilies : - str InstanceTypes : - str NodeCounts : DefaultMinCount : str DefaultMaxCount : str ComputeResourceCounts : str: # ComputeResourceName MinCount : int MaxCount : int AdditionalSecurityGroups : - str AdditionalIamPolicies : - str OnPremComputeNodes : ConfigFile : str CIDR : str Partition : str SlurmUid : int storage : ExtraMounts : - dest : str src : str type : str options : str StorageType : str FileSystemId : str VolumeId : str ExtraMountSecurityGroups : FileSystemType : SecurityGroupName: SecurityGroupId Licenses : LicenseName : Count : int Server : str Port : str ServerType : StatusScript :","title":"Configuraton File Format"},{"location":"config/#top-level-config","text":"","title":"Top Level Config"},{"location":"config/#termination_protection","text":"Enable Cloudformation Stack termination protection default=True","title":"termination_protection"},{"location":"config/#stackname","text":"The name of the configuration stack that will configure ParallelCluster and deploy it. If you do not specify the ClusterName then it will default to a value based on the StackName. If StackName ends in -config then ClusterName will be the StackName with -config stripped off. Otherwise it will be the StackName with -cl (for cluster) appended. Optional so can be specified on the command-line default='slurm-config'","title":"StackName"},{"location":"config/#region","text":"AWS region where the cluster will be deployed. Optional so can be specified on the command-line","title":"Region"},{"location":"config/#sshkeypair","text":"Default EC2 key pair that will be used for all cluster instances. Optional so can be specified on the command-line","title":"SshKeyPair"},{"location":"config/#vpcid","text":"The ID of the VPC where the cluster will be deployed. Optional so can be specified on the command-line","title":"VpcId"},{"location":"config/#cidr","text":"The CIDR of the VPC. This is used in security group rules.","title":"CIDR"},{"location":"config/#subnetid","text":"The ID of the VPC subnet where the cluster will be deployed. Optional. If not specified then the first private subnet is chosen. If no private subnets exist, then the first isolated subnet is chosen. If no isolated subnets exist, the the first public subnet is chosen. We recommend using a private or isolated subnet.","title":"SubnetId"},{"location":"config/#errorsnstopicarn","text":"The ARN of an existing SNS topic. Errors will be published to the SNS topic. You can subscribe to the topic so that you are notified for things like script or lambda errors. Optional, but highly recommended","title":"ErrorSnsTopicArn"},{"location":"config/#timezone","text":"The time zone to use for all EC2 instances in the cluster. default='US/Central'","title":"TimeZone"},{"location":"config/#resenvironmentname","text":"If you are deploying the cluster to use from Research and Engineering Studio (RES) virtual desktops, then you can specify the environment name so that the virtual desktops automatically get configured to use the cluster. The security group of the desktops will be updated with rules that allow them to talk to the cluster and the cluster will be configured on the desktop. The Slurm binaries will be compiled for the OS of the desktops and and environment modulefile will be created so that the users just need to load the cluster modulefile to use the cluster.","title":"RESEnvironmentName"},{"location":"config/#slurm","text":"Slurm configuration parameters.","title":"slurm"},{"location":"config/#parallelclusterconfig","text":"ParallelCluster specific configuration parameters.","title":"ParallelClusterConfig"},{"location":"config/#version","text":"The ParallelCluster version. This is required and cannot be changed after the cluster is created. Updating to a new version of ParallelCluster requires either deleting the current cluster or creating a new cluster.","title":"Version"},{"location":"config/#clusterconfig","text":"type: dict Additional ParallelCluster configuration settings that will be directly added to the configuration without checking. This will will be used to create the initial ParallelCluster configuration and other settings in this configuration file will override values in the dict. This exists to enable further customization of ParallelCluster beyond what this configuration supports.","title":"ClusterConfig"},{"location":"config/#image","text":"The OS and AMI to use for the head node and compute nodes.","title":"Image"},{"location":"config/#os","text":"See the ParallelCluster docs for the supported OS distributions and versions.","title":"OS"},{"location":"config/#customami","text":"See the ParallelCluster docs for the custom AMI documentation. NOTE : A CustomAmi must be provided for Rocky8 or Rocky9. All other distributions have a default AMI that is provided by ParallelCluster.","title":"CustomAmi"},{"location":"config/#architecture","text":"The CPU architecture to use for the cluster. ParallelCluster doesn't support heterogeneous clusters. All of the instances must have the same CPU architecture and the same OS. The cluster, however, can be accessed from login nodes of any architecture and OS. Valid Values: arm64 x86_64 default: x86_64","title":"Architecture"},{"location":"config/#computenodeami","text":"AMI to use for compute nodes. All compute nodes will use the same AMI. The default AMI is selected by the Image parameters.","title":"ComputeNodeAmi"},{"location":"config/#disablesimultaneousmultithreading","text":"type: bool default=True Disable SMT on the compute nodes. If true, multithreading on the compute nodes is disabled. Not all instance types can disable multithreading. For a list of instance types that support disabling multithreading, see CPU cores and threads for each CPU core per instance type in the Amazon EC2 User Guide for Linux Instances. Update policy: The compute fleet must be stopped for this setting to be changed for an update. ParallelCluster documentation","title":"DisableSimultaneousMultithreading"},{"location":"config/#enableefa","text":"type: bool default: False Recommend to not use EFA unless necessary to avoid insufficient capacity errors when starting new instances in group or when multiple instance types in the group. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-cluster","title":"EnableEfa"},{"location":"config/#database","text":"Optional Configure the Slurm database to use with the cluster. This is created independently of the cluster so that the same database can be used with multiple clusters. The easiest way to do this is to use the CloudFormation template provided by ParallelCluster and then to just pass the name of the stack in DatabaseStackName . All of the other parameters will be pulled from the stack. See the ParallelCluster documentation .","title":"Database"},{"location":"config/#databasestackname","text":"Name of the ParallelCluster CloudFormation stack that created the database. The following parameters will be set using the outputs of the stack: FQDN Port AdminUserName AdminPasswordSecretArn ClientSecurityGroup","title":"DatabaseStackName"},{"location":"config/#fqdn","text":"Used with the Port to set the Uri of the database.","title":"FQDN"},{"location":"config/#port","text":"type: int Database's port.","title":"Port"},{"location":"config/#adminusername","text":"type: str The identity that Slurm uses to connect to the database, write accounting logs, and perform queries. The user must have both read and write permissions on the database. Sets the UserName parameter in ParallelCluster.","title":"AdminUserName"},{"location":"config/#adminpasswordsecretarn","text":"type: str The Amazon Resource Name (ARN) of the AWS Secrets Manager secret that contains the AdminUserName plaintext password. This password is used together with AdminUserName and Slurm accounting to authenticate on the database server. Sets the PasswordSecretArn parameter in ParallelCluster.","title":"AdminPasswordSecretArn"},{"location":"config/#clientsecuritygroup","text":"Security group that has permissions to connect to the database. Required to be attached to the head node that is running slurmdbd so that the port connection to the database is allows.","title":"ClientSecurityGroup"},{"location":"config/#clustername","text":"Name of the ParallelCluster cluster. Default: If StackName ends with \"-config\" then ClusterName is StackName with \"-config\" stripped off. Otherwise add \"-cl\" to end of StackName.","title":"ClusterName"},{"location":"config/#mungekeysecret","text":"AWS secret with a base64 encoded munge key to use for the cluster. For an existing secret can be the secret name or the ARN. If the secret doesn't exist one will be created, but won't be part of the cloudformation stack so that it won't be deleted when the stack is deleted. Required if your submitters need to use more than 1 cluster.","title":"MungeKeySecret"},{"location":"config/#slurmctl","text":"Configure the Slurm head node or controller. Required, but can be an empty dict to accept all of the defaults.","title":"SlurmCtl"},{"location":"config/#slurmdport","text":"Port used for the slurmd daemon on the compute nodes. default=6818 type: int","title":"SlurmdPort"},{"location":"config/#instance_type","text":"Instance type of the head node. Must match the architecture of the cluster.","title":"instance_type"},{"location":"config/#volume_size","text":"The size of the EBS root volume on the head node in GB. default=200 type: int","title":"volume_size"},{"location":"config/#cloudwatchperiod","text":"The frequency of CloudWatch metrics in seconds. default=5 type: int","title":"CloudWatchPeriod"},{"location":"config/#preemptmode","text":"Set job preemption policy for the cluster. Jobs can be set to be preemptible when they are submitted. This allows higher priority jobs to preempt a running job when resources are constrained. This policy sets what happens to the preempted jobs. Slurm documentation Valid values: 'OFF' 'CANCEL' 'GANG' 'REQUEUE' 'SUSPEND' default='REQUEUE'","title":"PreemptMode"},{"location":"config/#preempttype","text":"Slurm documentation Valid values: 'preempt/none' 'preempt/partition_prio' 'preempt/qos' default='preempt/partition_prio'","title":"PreemptType"},{"location":"config/#preemptexempttime","text":"Slurm documentation Global option for minimum run time for all jobs before they can be considered for preemption. A time of -1 disables the option, equivalent to 0. Acceptable time formats include \"minutes\", \"minutes:seconds\", \"hours:minutes:seconds\", \"days-hours\", \"days-hours:minutes\", and \"days-hours:minutes:seconds\". default='0' type: str","title":"PreemptExemptTime"},{"location":"config/#slurmconfoverrides","text":"File that will be included at end of slurm.conf to override configuration parameters. This allows you to customize the slurm configuration arbitrarily. This should be used with caution since it can result in errors that make the cluster non-functional. type: str","title":"SlurmConfOverrides"},{"location":"config/#slurmrestduid","text":"User ID for the slurmrestd daemon. type: int default=901","title":"SlurmrestdUid"},{"location":"config/#slurmrestapiversion","text":"The REST API version. This is automatically set based on the Slurm version being used by the ParallelCluster version. type: str default: ''0.0.39'","title":"SlurmRestApiVersion"},{"location":"config/#head-node-additionalsecuritygroups","text":"Additional security groups that will be added to the head node instance.","title":"Head Node AdditionalSecurityGroups"},{"location":"config/#head-node-additionaliampolicies","text":"List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the head node instance.","title":"Head Node AdditionalIamPolicies"},{"location":"config/#submitterinstancetags","text":"Tags of instances that can be configured to submit to the cluster. When the cluster is deleted, the tag is used to unmount the slurm filesystem from the instances using SSM.","title":"SubmitterInstanceTags"},{"location":"config/#instanceconfig","text":"Configure the instances used by the cluster. A partition will be created for each combination of Base OS, Architecture, and Spot.","title":"InstanceConfig"},{"location":"config/#usespot","text":"Configure spot instances. type: bool default: True","title":"UseSpot"},{"location":"config/#exclude","text":"Instance families and types to exclude. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end.","title":"Exclude"},{"location":"config/#exclude-instancefamilies","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_families = [ 'a1', # Graviton 1 'c4', # Replaced by c5 'd2', # SSD optimized 'g3', # Replaced by g4 'g3s', # Replaced by g4 'h1', # SSD optimized 'i3', # SSD optimized 'i3en', # SSD optimized 'm4', # Replaced by m5 'p2', # Replaced by p3 'p3', 'p3dn', 'r4', # Replaced by r5 't2', # Replaced by t3 'x1', 'x1e', ]","title":"Exclude InstanceFamilies"},{"location":"config/#exclude-instancetypes","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_excluded_instance_types = [ '.+\\.(micro|nano)', # Not enough memory '.*\\.metal.*' ]","title":"Exclude InstanceTypes"},{"location":"config/#include","text":"Instance families and types to include. Exclude patterns are processed first and take precesdence over any includes. Instance families and types are regular expressions with implicit '^' and '$' at the begining and end.","title":"Include"},{"location":"config/#maxsizeonly","text":"type: bool default: False If MaxSizeOnly is True then only the largest instance type in a family will be included unless specific instance types are included.","title":"MaxSizeOnly"},{"location":"config/#include-instancefamilies","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_families = [ 'c7a', # AMD EPYC 9R14 Processor 3.7 GHz 'c7g', # AWS Graviton3 Processor 2.6 GHz # 'c7gd', # AWS Graviton3 Processor 2.6 GHz # 'c7gn', # AWS Graviton3 Processor 2.6 GHz # 'c7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz #'f1', # Intel Xeon E5-2686 v4 (Broadwell) 2.3 GHz 'm5zn', # Intel Xeon Platinum 8252 4.5 GHz 'm7a', # AMD EPYC 9R14 Processor 3.7 GHz # 'm7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'm7g', # AWS Graviton3 Processor 2.6 GHz # 'm7gd', # AWS Graviton3 Processor 2.6 GHz 'r7a', # AMD EPYC 9R14 Processor 3.7 GHz 'r7g', # AWS Graviton3 Processor 2.6 GHz # 'r7gd', # AWS Graviton3 Processor 2.6 GHz # 'r7i', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'r7iz', # Intel Xeon Scalable (Sapphire Rapids) 3.2 GHz 'x2gd', # AWS Graviton2 Processor 2.5 GHz 1TB 'x2idn', # Intel Xeon Scalable (Icelake) 3.5 GHz 2 TB 'x2iedn', # Intel Xeon Scalable (Icelake) 3.5 GHz 4 TB 'x2iezn', # Intel Xeon Platinum 8252 4.5 GHz 1.5 TB #'u-6tb1', # Intel Xeon Scalable (Skylake) 6 TB #'u-9tb1', # Intel Xeon Scalable (Skylake) 9 TB #'u-12tb1', # Intel Xeon Scalable (Skylake) 12 TB ]","title":"Include InstanceFamilies"},{"location":"config/#include-instancetypes","text":"Regular expressions with implicit '^' and '$' at the begining and end. An empty list is the same as '.*'. Default: default_eda_instance_types = [ #'c5\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz #'c5d\\.(l|x|2|4|9|18).*', # Intel Xeon Platinum 8124M 3.4 GHz #'c5d\\.(12|24).*', # Intel Xeon Platinum 8275L 3.6 GHz ]","title":"Include InstanceTypes"},{"location":"config/#nodecounts","text":"Configure the number of compute nodes of each instance type.","title":"NodeCounts"},{"location":"config/#defaultmincount","text":"type: int default: 0 Minimum number of compute nodes to keep running in a compute resource. If the number is greater than zero then static nodes will be created.","title":"DefaultMinCount"},{"location":"config/#defaultmaxcount","text":"type: int The maximum number of compute nodes to create in a compute resource.","title":"DefaultMaxCount"},{"location":"config/#computeresourcecounts","text":"Define compute node counts per compute resource. These counts will override the defaults set by DefaultMinCount and DefaultMaxCount .","title":"ComputeResourceCounts"},{"location":"config/#computeresourcename","text":"Name of the ParallelCluster compute resource. Can be found using sinfo .","title":"ComputeResourceName"},{"location":"config/#compute-resource-mincount","text":"type: int default: 0","title":"# Compute Resource MinCount"},{"location":"config/#compute-resource-maxcount","text":"type: int","title":"# Compute Resource MaxCount"},{"location":"config/#compute-node-additionalsecuritygroups","text":"Additional security groups that will be added to the compute node instances.","title":"Compute Node AdditionalSecurityGroups"},{"location":"config/#compute-node-additionaliampolicies","text":"List of Amazon Resource Names (ARNs) of IAM policies for Amazon EC2 that will be added to the compute node instances.","title":"Compute Node AdditionalIamPolicies"},{"location":"config/#onpremcomputenodes","text":"Define on-premises compute nodes that will be managed by the ParallelCluster head node. The compute nodes must be accessible from the head node over the network and any firewalls must allow all of the Slurm ports between the head node and compute nodes. ParallelCluster will be configured to allow the neccessary network traffic and the on-premises firewall can be configured to match the ParallelCluster seccurity groups.","title":"OnPremComputeNodes"},{"location":"config/#configfile","text":"Configuration file with the on-premises compute nodes defined in Slurm NodeName format as described in the Slurm slurm.conf documentation . The file will be included in the ParallelCluster slurm.conf so it can technically include any Slurm configuration updates including custom partition definitions. NOTE : The syntax of the file isn't checked and syntax errors can result in the slurmctld daemon failing on the head node.","title":"ConfigFile"},{"location":"config/#on-premises-cidr","text":"The CIDR that contains the on-premises compute nodes. This is to allow egress from the head node to the on-premises nodes.","title":"On-Premises CIDR"},{"location":"config/#partition","text":"A partition that will contain all of the on-premises nodes.","title":"Partition"},{"location":"config/#slurmuid","text":"type: int default: 900 The user id of the slurm user.","title":"SlurmUid"},{"location":"config/#storage","text":"","title":"storage"},{"location":"config/#extramounts","text":"Additional mounts for compute nodes. This can be used so the compute nodes have the same file structure as the remote desktops. This is used to configure ParallelCluster SharedStorage .","title":"ExtraMounts"},{"location":"config/#dest","text":"The directory where the file system will be mounted. This sets the MountDir .","title":"dest"},{"location":"config/#src","text":"The source path on the file system export that will be mounted.","title":"src"},{"location":"config/#type","text":"The type of mount. For example, nfs3.","title":"type"},{"location":"config/#options","text":"Mount options.","title":"options"},{"location":"config/#storagetype","text":"The type of file system to mount. Valid values: Efs FsxLustre FsxOntap FsxOpenZfs","title":"StorageType"},{"location":"config/#filesystemid","text":"Specifies the ID of an existing FSx for Lustre or EFS file system.","title":"FileSystemId"},{"location":"config/#volumeid","text":"Specifies the volume ID of an existing FSx for ONTAP or FSx for OpenZFS file system.","title":"VolumeId"},{"location":"config/#extramountsecuritygroups","text":"The security groups used by the file systems so that the head and comnpute nodes can be configured to connect to them. For example: storage: ExtraMounts: - dest: \"/tools\" StorageType: FsxOpenZfs VolumeId: 'fsvol-abcd1234' src: 'fs-efgh5678.fsx.us-east-1.amazonaws.com:/fsx/' type: nfs4 options: 'nfsvers=4.1' ExtraMountSecurityGroups: zfs: FsxSG: sg-12345678","title":"ExtraMountSecurityGroups"},{"location":"config/#filesystemtype","text":"Type of file system so that the appropriate ports can be opened. Valid values: nfs zfs lustre","title":"FileSystemType"},{"location":"config/#licenses","text":"Configure license counts for the scheduler. If the Slurm database is configured then it will be updated with the license counts. Otherwise, the license counts will be added to slurm.conf.","title":"Licenses"},{"location":"config/#licensename","text":"The name of the license, for example, VCSCompiler_Net or VCSMXRunTime_Net . This is the license name that users specify when submitting a job. It doesn't have to match the license name reported by the license server, although that probably makes the most sense.","title":"LicenseName"},{"location":"config/#count","text":"The number of licenses available to Slurm to use to schedule jobs. Once all of the license are used by running jobs, then any pending jobs will remain pending until a license becomes available.","title":"Count"},{"location":"config/#server","text":"The license server hosting the licenses. Not currently used.","title":"Server"},{"location":"config/#port_1","text":"The port on the license server used to request licenses. Not currently used.","title":"Port"},{"location":"config/#servertype","text":"The type of license server, such as FlexLM. Not currently used.","title":"ServerType"},{"location":"config/#statusscript","text":"A script that queries the license server and dynamically updates the Slurm database with the actual total number of licenses and the number used. Not currently implemented.","title":"StatusScript"},{"location":"custom-amis/","text":"Custom AMIs for ParallelCluster ParallelCluster supports building custom ParallelCluster AMIs for the head and compute nodes . You can specify a custom AMI for the entire cluster (head and compute nodes) and you can also specify a custom AMI for just the compute nodes. By default, ParallelCluster will use pre-built AMIs for the OS that you select. The exception is Rocky 8 and 9, for which ParallelCluster does not provide pre-built AMIs. To use Rocky Linux, you must first build a custom AMI and specify it in your config file at slurm/ParallelClusterConfig/Os/CustomAmi . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The build files can be modified for your needs. The build file format is docummented in the ParallelCluster User Guide . For example, you can add your own scripts to run during the AMI build process. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete. FPGA Developer AMI The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 . Deploy or update the Cluster After the AMI is built, add it to the config and create or update your cluster to use the AMI. You can set the AMI for the compute and head nodes using slurm/ParallelClusterConfig/Os/CustomAmi and for the compute nodes only using slurm/ParallelClusterConfig/ComputeNodeAmi . Note : You cannot update the OS of the cluster or the AMI of the head node. If they need to change then you will need to create a new cluster. The config file will look something like the following: slurm: ParallelClusterConfig: Image: Os: rocky8 CustomAmi: ami-abc123 # Rocky linux ComputeNodeAmi: ami-def456 # Rocky linux + EDA packages and EDA file systems","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#custom-amis-for-parallelcluster","text":"ParallelCluster supports building custom ParallelCluster AMIs for the head and compute nodes . You can specify a custom AMI for the entire cluster (head and compute nodes) and you can also specify a custom AMI for just the compute nodes. By default, ParallelCluster will use pre-built AMIs for the OS that you select. The exception is Rocky 8 and 9, for which ParallelCluster does not provide pre-built AMIs. To use Rocky Linux, you must first build a custom AMI and specify it in your config file at slurm/ParallelClusterConfig/Os/CustomAmi . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The build files can be modified for your needs. The build file format is docummented in the ParallelCluster User Guide . For example, you can add your own scripts to run during the AMI build process. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete.","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#fpga-developer-ami","text":"The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 .","title":"FPGA Developer AMI"},{"location":"custom-amis/#deploy-or-update-the-cluster","text":"After the AMI is built, add it to the config and create or update your cluster to use the AMI. You can set the AMI for the compute and head nodes using slurm/ParallelClusterConfig/Os/CustomAmi and for the compute nodes only using slurm/ParallelClusterConfig/ComputeNodeAmi . Note : You cannot update the OS of the cluster or the AMI of the head node. If they need to change then you will need to create a new cluster. The config file will look something like the following: slurm: ParallelClusterConfig: Image: Os: rocky8 CustomAmi: ami-abc123 # Rocky linux ComputeNodeAmi: ami-def456 # Rocky linux + EDA packages and EDA file systems","title":"Deploy or update the Cluster"},{"location":"debug/","text":"Debug For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation . Slurm Head Node If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv Compute Nodes If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd Log Files Logfile Description /var/log/slurmd.log slurmctld logfile Job Stuck in Pending State You can use scontrol to get detailed information about a job. scontrol show job *jobid* Job Stuck in Completing State When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Debug"},{"location":"debug/#debug","text":"For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation .","title":"Debug"},{"location":"debug/#slurm-head-node","text":"If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv","title":"Slurm Head Node"},{"location":"debug/#compute-nodes","text":"If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd","title":"Compute Nodes"},{"location":"debug/#log-files","text":"Logfile Description /var/log/slurmd.log slurmctld logfile","title":"Log Files"},{"location":"debug/#job-stuck-in-pending-state","text":"You can use scontrol to get detailed information about a job. scontrol show job *jobid*","title":"Job Stuck in Pending State"},{"location":"debug/#job-stuck-in-completing-state","text":"When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Job Stuck in Completing State"},{"location":"delete-cluster/","text":"Delete Cluster To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"delete-cluster/#delete-cluster","text":"To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"deploy-parallel-cluster/","text":"Deploy AWS ParallelCluster A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0. Prerequisites See Deployment Prerequisites page. Create ParallelCluster UI (optional but recommended) It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide . Create ParallelCluster Slurm Database The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting . Create the Cluster To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster. Create users_groups.json Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys. Configure submission hosts to use the cluster ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm. Customize the compute node AMI The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again. Run Your First Job Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash Slurm Documentation https://slurm.schedmd.com","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#deploy-aws-parallelcluster","text":"A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0.","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#prerequisites","text":"See Deployment Prerequisites page.","title":"Prerequisites"},{"location":"deploy-parallel-cluster/#create-parallelcluster-ui-optional-but-recommended","text":"It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide .","title":"Create ParallelCluster UI (optional but recommended)"},{"location":"deploy-parallel-cluster/#create-parallelcluster-slurm-database","text":"The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting .","title":"Create ParallelCluster Slurm Database"},{"location":"deploy-parallel-cluster/#create-the-cluster","text":"To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster.","title":"Create the Cluster"},{"location":"deploy-parallel-cluster/#create-users_groupsjson","text":"Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys.","title":"Create users_groups.json"},{"location":"deploy-parallel-cluster/#configure-submission-hosts-to-use-the-cluster","text":"ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm.","title":"Configure submission hosts to use the cluster"},{"location":"deploy-parallel-cluster/#customize-the-compute-node-ami","text":"The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again.","title":"Customize the compute node AMI"},{"location":"deploy-parallel-cluster/#run-your-first-job","text":"Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash","title":"Run Your First Job"},{"location":"deploy-parallel-cluster/#slurm-documentation","text":"https://slurm.schedmd.com","title":"Slurm Documentation"},{"location":"deployment-prerequisites/","text":"Deployment Prerequisites This page shows common prerequisites that need to be done before deployment. Deployment Server/Instance Requirements The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance. Configure AWS CLI Credentials You will needs AWS credentials that provide admin access to deploy the cluster. Clone or Download the Repository Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git Create SNS Topic for Error Notifications (Optional but recommended) The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket. Make sure using at least python version 3.7 This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5 Make sure required packages are installed cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment. Install Cloud Development Kit (CDK) (Optional) The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed. Security Groups for Login Nodes If you want to allow instances like remote desktops to use the cluster directly, you must define three security groups that allow connections between the instance, the Slurm head node, and the Slurm compute nodes. We call the instance that is connecting to the Slurm cluster a login node or a submitter instance. I'll call the three security groups the following names, but they can be whatever you want. SlurmSubmitterSG SlurmHeadNodeSG SlurmComputeNodeSG Slurm Submitter Security Group The SlurmSubmitterSG will be attached to your login nodes, such as your virtual desktops. It needs at least the following inbound rules: Type Port range Source Description TCP 1024-65535 SlurmHeadNodeSG SlurmHeadNode ephemeral TCP 1024-65535 SlurmComputeNodeSG SlurmComputeNode ephemeral TCP 6000-7024 SlurmComputeNodeSG SlurmComputeNode X11 It needs the following outbound rules. Type Port range Destination Description TCP 2049 SlurmHeadNodeSG SlurmHeadNode NFS TCP 6818 SlurmComputeNodeSG SlurmComputeNode slurmd TCP 6819 SlurmHeadNodeSG SlurmHeadNode slurmdbd TCP 6820-6829 SlurmHeadNodeSG SlurmHeadNode slurmctld TCP 6830 SlurmHeadNodeSG SlurmHeadNode slurmrestd Slurm Head Node Security Group The SlurmHeadNodeSG will be specified in your configuration file for the slurm/SlurmCtl/AdditionalSecurityGroups parameter. It needs at least the following inbound rules: Type Port range Source Description TCP 2049 SlurmSubmitterSG SlurmSubmitter NFS TCP 6819 SlurmSubmitterSG SlurmSubmitter slurmdbd TCP 6820-6829 SlurmSubmitterSG SlurmSubmitter slurmctld TCP 6830 SlurmSubmitterSG SlurmSubmitter slurmrestd It needs the following outbound rules. Type Port range Destination Description TCP 1024-65535 SlurmSubmitterSG SlurmSubmitter ephemeral Slurm Compute Node Security Group The SlurmComputeNodeSG will be specified in your configuration file for the slurm/InstanceConfig/AdditionalSecurityGroups parameter. It needs at least the following inbound rules: Type Port range Source Description TCP 6818 SlurmSubmitterSG SlurmSubmitter slurmd It needs the following outbound rules. Type Port range Destination Description TCP 1024-65535 SlurmSubmitterSG SlurmSubmitter ephemeral TCP 6000-7024 SlurmSubmitterSG SlurmSubmitter X11 Create Configuration File Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. See Configuration File Format for documentation on all of the parameters. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml Configure the Compute Instances The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 RedHat 9 x86_64, arm64 Rocky 8 x86_64, arm64 Rocky 9 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1 Configure Fair Share Scheduling (Optional) Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities. Configure Licenses Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-prerequisites","text":"This page shows common prerequisites that need to be done before deployment.","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-serverinstance-requirements","text":"The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance.","title":"Deployment Server/Instance Requirements"},{"location":"deployment-prerequisites/#configure-aws-cli-credentials","text":"You will needs AWS credentials that provide admin access to deploy the cluster.","title":"Configure AWS CLI Credentials"},{"location":"deployment-prerequisites/#clone-or-download-the-repository","text":"Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git","title":"Clone or Download the Repository"},{"location":"deployment-prerequisites/#create-sns-topic-for-error-notifications-optional-but-recommended","text":"The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket.","title":"Create SNS Topic for Error Notifications (Optional but recommended)"},{"location":"deployment-prerequisites/#make-sure-using-at-least-python-version-37","text":"This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5","title":"Make sure using at least python version 3.7"},{"location":"deployment-prerequisites/#make-sure-required-packages-are-installed","text":"cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment.","title":"Make sure required packages are installed"},{"location":"deployment-prerequisites/#install-cloud-development-kit-cdk-optional","text":"The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed.","title":"Install Cloud Development Kit (CDK) (Optional)"},{"location":"deployment-prerequisites/#security-groups-for-login-nodes","text":"If you want to allow instances like remote desktops to use the cluster directly, you must define three security groups that allow connections between the instance, the Slurm head node, and the Slurm compute nodes. We call the instance that is connecting to the Slurm cluster a login node or a submitter instance. I'll call the three security groups the following names, but they can be whatever you want. SlurmSubmitterSG SlurmHeadNodeSG SlurmComputeNodeSG","title":"Security Groups for Login Nodes"},{"location":"deployment-prerequisites/#slurm-submitter-security-group","text":"The SlurmSubmitterSG will be attached to your login nodes, such as your virtual desktops. It needs at least the following inbound rules: Type Port range Source Description TCP 1024-65535 SlurmHeadNodeSG SlurmHeadNode ephemeral TCP 1024-65535 SlurmComputeNodeSG SlurmComputeNode ephemeral TCP 6000-7024 SlurmComputeNodeSG SlurmComputeNode X11 It needs the following outbound rules. Type Port range Destination Description TCP 2049 SlurmHeadNodeSG SlurmHeadNode NFS TCP 6818 SlurmComputeNodeSG SlurmComputeNode slurmd TCP 6819 SlurmHeadNodeSG SlurmHeadNode slurmdbd TCP 6820-6829 SlurmHeadNodeSG SlurmHeadNode slurmctld TCP 6830 SlurmHeadNodeSG SlurmHeadNode slurmrestd","title":"Slurm Submitter Security Group"},{"location":"deployment-prerequisites/#slurm-head-node-security-group","text":"The SlurmHeadNodeSG will be specified in your configuration file for the slurm/SlurmCtl/AdditionalSecurityGroups parameter. It needs at least the following inbound rules: Type Port range Source Description TCP 2049 SlurmSubmitterSG SlurmSubmitter NFS TCP 6819 SlurmSubmitterSG SlurmSubmitter slurmdbd TCP 6820-6829 SlurmSubmitterSG SlurmSubmitter slurmctld TCP 6830 SlurmSubmitterSG SlurmSubmitter slurmrestd It needs the following outbound rules. Type Port range Destination Description TCP 1024-65535 SlurmSubmitterSG SlurmSubmitter ephemeral","title":"Slurm Head Node Security Group"},{"location":"deployment-prerequisites/#slurm-compute-node-security-group","text":"The SlurmComputeNodeSG will be specified in your configuration file for the slurm/InstanceConfig/AdditionalSecurityGroups parameter. It needs at least the following inbound rules: Type Port range Source Description TCP 6818 SlurmSubmitterSG SlurmSubmitter slurmd It needs the following outbound rules. Type Port range Destination Description TCP 1024-65535 SlurmSubmitterSG SlurmSubmitter ephemeral TCP 6000-7024 SlurmSubmitterSG SlurmSubmitter X11","title":"Slurm Compute Node Security Group"},{"location":"deployment-prerequisites/#create-configuration-file","text":"Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. See Configuration File Format for documentation on all of the parameters. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml","title":"Create Configuration File"},{"location":"deployment-prerequisites/#configure-the-compute-instances","text":"The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 RedHat 9 x86_64, arm64 Rocky 8 x86_64, arm64 Rocky 9 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1","title":"Configure the Compute Instances"},{"location":"deployment-prerequisites/#configure-fair-share-scheduling-optional","text":"Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities.","title":"Configure Fair Share Scheduling (Optional)"},{"location":"deployment-prerequisites/#configure-licenses","text":"Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Configure Licenses"},{"location":"federation/","text":"Federation (legacy) To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"federation/#federation-legacy","text":"To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"implementation/","text":"Implementation Details (legacy) Slurm Infrastructure All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons. Directory Structure All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}} CloudWatch Metrics CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py Down Node Handling If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer Insufficient Capacity Exception (ICE) Handling When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Implementation Details (legacy)"},{"location":"implementation/#implementation-details-legacy","text":"","title":"Implementation Details (legacy)"},{"location":"implementation/#slurm-infrastructure","text":"All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons.","title":"Slurm Infrastructure"},{"location":"implementation/#directory-structure","text":"All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}}","title":"Directory Structure"},{"location":"implementation/#cloudwatch-metrics","text":"CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py","title":"CloudWatch Metrics"},{"location":"implementation/#down-node-handling","text":"If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer","title":"Down Node Handling"},{"location":"implementation/#insufficient-capacity-exception-ice-handling","text":"When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Insufficient Capacity Exception (ICE) Handling"},{"location":"job_preemption/","text":"Job Preemption The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license. Documentation https://slurm.schedmd.com/preempt.html","title":"Job Preemption"},{"location":"job_preemption/#job-preemption","text":"The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license.","title":"Job Preemption"},{"location":"job_preemption/#documentation","text":"https://slurm.schedmd.com/preempt.html","title":"Documentation"},{"location":"onprem/","text":"On-Premises Integration The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information. Network Requirements The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes. DNS Requirements Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC. File System Requirements All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm. Slurm Configuration of On-Premises Compute Nodes The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem Simulating an On-Premises Network Using AWS Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"On-Premises Integration"},{"location":"onprem/#on-premises-integration","text":"The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information.","title":"On-Premises Integration"},{"location":"onprem/#network-requirements","text":"The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes.","title":"Network Requirements"},{"location":"onprem/#dns-requirements","text":"Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC.","title":"DNS Requirements"},{"location":"onprem/#file-system-requirements","text":"All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm.","title":"File System Requirements"},{"location":"onprem/#slurm-configuration-of-on-premises-compute-nodes","text":"The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem","title":"Slurm Configuration of On-Premises Compute Nodes"},{"location":"onprem/#simulating-an-on-premises-network-using-aws","text":"Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"Simulating an On-Premises Network Using AWS"},{"location":"res_integration/","text":"RES Integration Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx slurm/SlurmCtl/AdditionalSecurityGroups Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. slurm/InstanceConfig/AdditionalSecurityGroups Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"res_integration/#res-integration","text":"Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx slurm/SlurmCtl/AdditionalSecurityGroups Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. slurm/InstanceConfig/AdditionalSecurityGroups Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"rest_api/","text":"Slurm REST API The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster. How to use the REST API The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"Slurm REST API"},{"location":"rest_api/#slurm-rest-api","text":"The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster.","title":"Slurm REST API"},{"location":"rest_api/#how-to-use-the-rest-api","text":"The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"How to use the REST API"},{"location":"run_jobs/","text":"Run Jobs This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages. Set Up Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format. Key Slurm Commands The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state sbatch The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script. Run a simulation build followed by a regression build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi srun The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs squeue The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue sprio Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11 sacct Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct sreport The sreport command can be used to generate report from the Slurm database. Other Slurm Commands Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Run Jobs"},{"location":"run_jobs/#run-jobs","text":"This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages.","title":"Run Jobs"},{"location":"run_jobs/#set-up","text":"Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format.","title":"Set Up"},{"location":"run_jobs/#key-slurm-commands","text":"The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state","title":"Key Slurm Commands"},{"location":"run_jobs/#sbatch","text":"The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script.","title":"sbatch"},{"location":"run_jobs/#run-a-simulation-build-followed-by-a-regression","text":"build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi","title":"Run a simulation build followed by a regression"},{"location":"run_jobs/#srun","text":"The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs","title":"srun"},{"location":"run_jobs/#squeue","text":"The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue","title":"squeue"},{"location":"run_jobs/#sprio","text":"Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11","title":"sprio"},{"location":"run_jobs/#sacct","text":"Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct","title":"sacct"},{"location":"run_jobs/#sreport","text":"The sreport command can be used to generate report from the Slurm database.","title":"sreport"},{"location":"run_jobs/#other-slurm-commands","text":"Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Other Slurm Commands"},{"location":"soca_integration/","text":"SOCA Integration Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx slurm/SlurmCtl/AdditionalSecurityGroups Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. slurm/InstanceConfig/AdditionalSecurityGroups Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"},{"location":"soca_integration/#soca-integration","text":"Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx slurm/SlurmCtl/AdditionalSecurityGroups Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. slurm/InstanceConfig/AdditionalSecurityGroups Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 5abd43d8..bef18bb6 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
diff --git a/soca_integration/index.html b/soca_integration/index.html
index e32f1ae1..be749fe0 100644
--- a/soca_integration/index.html
+++ b/soca_integration/index.html
@@ -140,9 +140,14 @@ SOCA Integration
vpc-xxxxxx |
-| SubmitterSecurityGroupIds |
-The ComputeNode security group name and id |
-cluster-id-ComputeNodeSG: sg-xxxxxxxx |
+slurm/SlurmCtl/AdditionalSecurityGroups |
+Security group ids that give desktop instances access to the head node and that give the head node access to VPC resources such as file systems. |
+ |
+
+
+| slurm/InstanceConfig/AdditionalSecurityGroups |
+Security group ids that give desktop instances access to the compute nodes and that give compute nodes access to VPC resources such as file systems. |
+ |
| ExtraMounts |