Add utility scripts to simplify testing of a new OS #2328

enrico-usai · 2023-06-26T08:25:54Z

Add new utility os-resources.py to simplify creation of OS specific resources

The new utility os-resources.py script is able to copy the resources/<name>/<name>_{src_platform}{src_version}
into a resources/<name>/<name>_{dst_platform}{dst_version}, replacing the following patterns:

platform: '{src_platform}'
platform?('{src_platform}')
to_i == {src_version}
platform_version: '{src_version}'

This utility permits to automatically generate all the required files,
then they must be checked one by one to see if there are OS specific checks on them.
Templates files must be manually copied.

Add test-os.sh utility to simplify run of kitchen tests for an OS

The script is able to search for all the test suites defined
into the functional cookbook and run them sequentially.

It must be executed from the root folder of the cookbook repo:
bash util/test-os.sh ec2 rhel8

The output will be saved into a /tmp/kitchen-<os>-<date> folder.

To run config tests is required to set a KITCHEN_${os}_AMI environment variable (e.g. KITCHEN_RHEL8_AMI)
or have a ParallelCluster AMI in the account.

codecov · 2023-06-26T08:27:26Z

Codecov Report

Merging #2328 (dbe2e8a) into develop (833a541) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #2328   +/-   ##
========================================
  Coverage    70.16%   70.16%           
========================================
  Files           13       13           
  Lines         1840     1840           
========================================
  Hits          1291     1291           
  Misses         549      549

Flag	Coverage Δ
unittests	`70.16% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

util/os-resources.py

… resources The new utility `os-resources.py` script is able to copy the `resources/<name>/<name>_{src_platform}{src_version}` into a `resources/<name>/<name>_{dst_platform}{dst_version}`, replacing the following patterns: * `platform: '{src_platform}'` * `platform?('{src_platform}')`` * `to_i == {src_version}`` * `platform_version: '{src_version}'` This utility permits to automatically generate all the required files, then they must be checked one by one to see if there are OS specific checks on them. Templates and other specific files must be manually copied. Signed-off-by: Enrico Usai <[email protected]>

The script is able to search for all the test suites defined into the functional cookbook and run them sequentially. It must be executed from the root folder of the cookbook repo: bash util/test-os.sh ec2 rhel8 The output will be saved into a `/tmp/kitchen-<os>-<date>` folder. To run config tests is required to set a KITCHEN_${os}_AMI environment variable (e.g. KITCHEN_RHEL8_AMI) or have a ParallelCluster AMI in the account. Signed-off-by: Enrico Usai <[email protected]>

### Description of changes Used new utility `os-resources.py` introduced as part of aws#2328 to create new resources for alinux2023, starting from redhat8 resources. Relevant changes to the code: * os_type --> Replaced rhel with alinux * Tried to fix CloudWatch agent setup by changing `platform_url_component` to point to the same of rhel (not sure if it's correct) * Copied network setup templates from redhat folders * Added alinux2023 to pcluster_dcv_connect.sh script Removed redhat_on_docker condition from: * stunnel * system_authentication * efa TODO: * efa -> check efa_supported? condition and log messages * lustre -> check version condition and log messages * install_packages --> Removed postgresql packages * Enable repository needed by hwloc-devel blas-devel libedit-devel and glibc-static packages ### Tests * Added Alinux2023 to ec2 kitchen configuration files. Copied from rhel8 with minor changes: * AMI name prefix took from EC2 Amazon Linux official AMI * I called the suite `alinux-2023`, with an "-" in the name to avoid having alinux2 as prefix of alinux-2023 and be able to distinguish them on Inspec runs. * Created new `pre_converge` hook to install libxcrypt-compat package, required to install Chef, leveraging the work done with aws#2342 * I had to pass a custom `provisioner/download_url` for cinc because package for AL2023 is not available in the default path. * The validated resources are: * nvidia_driver * arm_Pl * c_states * stunnel * build_tools * chrony * modules * munge * dns_domain (install only) * jwt_dependencies * nfs (install only) * raid (install only) * system_authentication (install only) * efs (install only) TODO: * Add Alinux2023 to GitHub actions * Add new os to kitchen.docker.yml config file (search for `kernel_release` version from an EC2 instance) * Fix Inspec and ChefSpec tests conditions ### References * https://hub.docker.com/_/amazonlinux * https://github.com/test-kitchen/kitchen-ec2/tree/main/lib/kitchen/driver/aws/standard_platform Known issues/FE: * amazonlinux/amazon-linux-2023#47 * amazonlinux/amazon-linux-2023#146 * amazonlinux/amazon-linux-2023#168 * amazonlinux/amazon-linux-2023#309 * amazonlinux/amazon-linux-2023#316 Signed-off-by: Enrico Usai <[email protected]>

### Description of changes Used new utility `os-resources.py` introduced as part of aws#2328 to create new resources for alinux2023, starting from redhat8 resources. Relevant changes to the code: * os_type --> Replaced rhel with alinux * Tried to fix CloudWatch agent setup by changing `platform_url_component` to point to the same of rhel (not sure if it's correct) * Copied network setup templates from redhat folders * Added alinux2023 to pcluster_dcv_connect.sh script Removed redhat_on_docker condition from: * stunnel * system_authentication * efa TODO: * efa -> check efa_supported? condition and log messages * lustre -> check version condition and log messages * install_packages --> Removed postgresql packages * Enable repository needed by hwloc-devel blas-devel libedit-devel and glibc-static packages ### Tests * Added Alinux2023 to ec2 kitchen configuration files. Copied from rhel8 with minor changes: * AMI name prefix took from EC2 Amazon Linux official AMI * I called the suite `alinux-2023`, with an "-" in the name to avoid having alinux2 as prefix of alinux-2023 and be able to distinguish them on Inspec runs. * Created new `pre_converge` hook to install libxcrypt-compat package, required to install Chef, leveraging the work done with aws#2342 * I had to pass a custom `provisioner/download_url` for cinc because package for AL2023 is not available in the default path. * The validated resources are: * nvidia_driver * arm_Pl * c_states * stunnel * build_tools * chrony * modules * munge * dns_domain (install only) * jwt_dependencies * nfs (install only) * raid (install only) * system_authentication (install only) * efs (install only) TODO: * Add Alinux2023 to GitHub actions * Add new os to kitchen.docker.yml config file (search for `kernel_release` version from an EC2 instance) * Fix Inspec and ChefSpec tests conditions ### References * https://hub.docker.com/_/amazonlinux * https://github.com/test-kitchen/kitchen-ec2/tree/main/lib/kitchen/driver/aws/standard_platform Known issues/FE: * amazonlinux/amazon-linux-2023#47 * amazonlinux/amazon-linux-2023#146 * amazonlinux/amazon-linux-2023#168 * amazonlinux/amazon-linux-2023#309 * amazonlinux/amazon-linux-2023#316 Signed-off-by: Enrico Usai <[email protected]> [AL2023] Use systemd-resolved instead of dhclient on Alinux 2023 Signed-off-by: Hanwen <[email protected]>

### Description of changes Used new utility `os-resources.py` introduced as part of #2328 to create new resources for alinux2023, starting from redhat8 resources. Relevant changes to the code: * os_type --> Replaced rhel with alinux * Tried to fix CloudWatch agent setup by changing `platform_url_component` to point to the same of rhel (not sure if it's correct) * Copied network setup templates from redhat folders * Added alinux2023 to pcluster_dcv_connect.sh script Removed redhat_on_docker condition from: * stunnel * system_authentication * efa TODO: * efa -> check efa_supported? condition and log messages * lustre -> check version condition and log messages * install_packages --> Removed postgresql packages * Enable repository needed by hwloc-devel blas-devel libedit-devel and glibc-static packages ### Tests * Added Alinux2023 to ec2 kitchen configuration files. Copied from rhel8 with minor changes: * AMI name prefix took from EC2 Amazon Linux official AMI * I called the suite `alinux-2023`, with an "-" in the name to avoid having alinux2 as prefix of alinux-2023 and be able to distinguish them on Inspec runs. * Created new `pre_converge` hook to install libxcrypt-compat package, required to install Chef, leveraging the work done with #2342 * I had to pass a custom `provisioner/download_url` for cinc because package for AL2023 is not available in the default path. * The validated resources are: * nvidia_driver * arm_Pl * c_states * stunnel * build_tools * chrony * modules * munge * dns_domain (install only) * jwt_dependencies * nfs (install only) * raid (install only) * system_authentication (install only) * efs (install only) TODO: * Add Alinux2023 to GitHub actions * Add new os to kitchen.docker.yml config file (search for `kernel_release` version from an EC2 instance) * Fix Inspec and ChefSpec tests conditions ### References * https://hub.docker.com/_/amazonlinux * https://github.com/test-kitchen/kitchen-ec2/tree/main/lib/kitchen/driver/aws/standard_platform Known issues/FE: * amazonlinux/amazon-linux-2023#47 * amazonlinux/amazon-linux-2023#146 * amazonlinux/amazon-linux-2023#168 * amazonlinux/amazon-linux-2023#309 * amazonlinux/amazon-linux-2023#316 Signed-off-by: Enrico Usai <[email protected]> [AL2023] Use systemd-resolved instead of dhclient on Alinux 2023 Signed-off-by: Hanwen <[email protected]>

enrico-usai added skip-changelog-update skip-system-tests labels Jun 26, 2023

enrico-usai requested review from a team as code owners June 26, 2023 08:25

gmarciani previously approved these changes Jun 26, 2023

View reviewed changes

util/os-resources.py Outdated Show resolved Hide resolved

util/os-resources.py Show resolved Hide resolved

util/os-resources.py Show resolved Hide resolved

enrico-usai dismissed gmarciani’s stale review via 2ba5175 June 26, 2023 10:39

enrico-usai force-pushed the wip/new-os branch from 297631a to 2ba5175 Compare June 26, 2023 10:39

enrico-usai added 2 commits June 26, 2023 12:43

enrico-usai force-pushed the wip/new-os branch from 2ba5175 to dbe2e8a Compare June 26, 2023 10:43

gmarciani approved these changes Jun 26, 2023

View reviewed changes

enrico-usai merged commit 68f1702 into aws:develop Jun 26, 2023

enrico-usai deleted the wip/new-os branch June 26, 2023 10:51

enrico-usai mentioned this pull request Jun 26, 2023

[POC] Add support for Rocky Linux 8 #2324

Merged

enrico-usai mentioned this pull request Jun 29, 2023

[POC] Add support for Amazon Linux 2023 #2343

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add utility scripts to simplify testing of a new OS #2328

Add utility scripts to simplify testing of a new OS #2328

enrico-usai commented Jun 26, 2023 •

edited

Loading

codecov bot commented Jun 26, 2023 •

edited

Loading

Add utility scripts to simplify testing of a new OS #2328

Add utility scripts to simplify testing of a new OS #2328

Conversation

enrico-usai commented Jun 26, 2023 • edited Loading

Add new utility os-resources.py to simplify creation of OS specific resources

Add test-os.sh utility to simplify run of kitchen tests for an OS

codecov bot commented Jun 26, 2023 • edited Loading

Codecov Report

enrico-usai commented Jun 26, 2023 •

edited

Loading

codecov bot commented Jun 26, 2023 •

edited

Loading