Computational clusters: when the EC2 vCPU quota is reached the clusters does not start processing

[sanderegg]

Long Story Short

A simulation that requires >=10 machines of c6a.24xlarge does not process.

Expected Behavior

The simulation should complete.
When the number of requested machines is above the current quota, it should try to reduce the number until it manages to create a machine.
Feedback to the user should be given (this is yet another feature)

Actual behaviour

• the computational cluster autoscaling service requests 1 <= X <= EC2_INSTANCES_MAX_INSTANCES EC2 instances to run the computational jobs
• if X * vCPUs is above the AWS EC2 quota assigned, then an error is returned and the autoscaling does not try with smaller number of machines.
  --> the user has no feedback
  --> the computation does not run at all
  --> it needs developers interventions in a relatively deep way (access to cluster, change of EC2_INSTANCES_MAX_INSTANCES variable to a lower number to unlock the computation)

Steps to reproduce

run multiple computational jobs that require a EC2 instance where the quota is almost at 100%
Environment

Additional context

[sanderegg]

Requested 5000vCPUs for C/R/T on-demand
Requested 5000vCPUs for G