multiple custom actions validators

[psacc]

Description of changes

• add support for Resources async validation in addition to sync validation
• add support for async validators timeout, caching and retry
• UrlValidators use async execution with caching and retry
• Multiple custom actions resources uses async UrlValidators with timeout (and caching + retry)

Tests

• manual local test: cluster creation with multiple custom actions
• added coverage for multiple custom actions in tests/integration-tests/tests/pcluster_api/test_api.py (testing async multithreaded validators in lambda)
• local integration tests execution:

test-suites:
  create:
    test_create.py::test_cluster_creation_with_problematic_preinstall_script:
      dimensions:
        - regions: ["eu-west-1"]
          instances: {{ common.INSTANCES_DEFAULT_ARM }}
          schedulers: ["slurm"]
          oss: ["alinux2"]
  cfn-init:
    test_cfn_init.py::test_replace_compute_on_failure:
      dimensions:
        - regions: ["eu-west-1"]
          instances: {{ common.INSTANCES_DEFAULT_X86 }}
          oss: ["alinux2"]
          schedulers: ["slurm"]
    test_cfn_init.py::test_install_args_quotes:
      dimensions:
        - regions: ["eu-west-1"]
          instances: {{ common.INSTANCES_DEFAULT_X86 }}
          oss: ["alinux2"]
          schedulers: ["slurm"]
  pcluster_api:
    test_api_infrastructure.py::test_api_infrastructure_with_default_parameters:
      dimensions:
        - regions: ["eu-west-1"]
    test_api.py::test_cluster_slurm:
      dimensions:
        - regions: ["eu-west-1"]
          instances: {{ common.INSTANCES_DEFAULT_X86 }}
          oss: ["alinux2"]
          schedulers: ["slurm"]
    test_api.py::test_cluster_awsbatch:
      dimensions:
        - regions: ["eu-west-1"]
          instances: {{ common.INSTANCES_DEFAULT_X86 }}
          oss: ["alinux2"]
          schedulers: ["awsbatch"]

References

• multiple custom actions support in yaml schema #5104
• fetch_and_run in python supporting multiple custom actions aws-parallelcluster-cookbook#1895
• fetch_and_run source /etc/parallelcluster/cfnconfig in a backward compatible way aws-parallelcluster-cookbook#1990

Checklist

• Make sure you are pointing to the right branch and add a label in the PR title (i.e. 2.x vs 3.x)
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

Merging #5134 (b4899ad) into develop (0704826) will increase coverage by 0.07%.
The diff coverage is n/a.

❗ Current head b4899ad differs from pull request most recent head b6797a0. Consider uploading reports for the commit b6797a0 to get more accurate results

@@             Coverage Diff             @@
##           develop    #5134      +/-   ##
===========================================
+ Coverage    89.65%   89.72%   +0.07%     
===========================================
  Files          175      175              
  Lines        15131    15241     +110     
===========================================
+ Hits         13565    13675     +110     
  Misses        1566     1566              

Flag	Coverage Δ
unittests	89.72% <ø> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 4 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[eantonin]

LGTM, great work

[eantonin]

nitpick, non-blocking: This method and some of the below ones are copied from another class, we may put them under the generic conftest.py to DRY

[eantonin]

nitpick,non-blocking: you can just hardcode 0 in here

Or, if you want to have just the half of the https url with cache, this does not work

[psacc]

good catch, introduced after adding s3 urls, fixed