✨Computational autoscaling: find out which EC2 type is necessary

[sanderegg]

What do these changes do?

This PR shall allow the autoscaling service to find out which EC2 type is desired by looking into the task resources.

New

• If a specific EC2 instance type is defined then autoscaling will pick it up, and if this is in accordance with EC2_INSTANCES_ALLOWED_TYPES then it will start such a machine instead of fitting it to the task needed resources. In case no specific EC2 instance type is defined, then the usual best fitting machine algorithm is used.
• the created nodes are now also tagged with their respective EC2 type: e.g. a node label such as ec2-instance-type=t2.micro is set on each autoscaled node

Computational mode:

• a Dask task specific EC2 instance type is defined as a "task restriction", named EC2-INSTANCE-TYPE

Dynamic mode:

• a Docker service may define a "Placement Constraint" such as ```node.labels.ec2-instance-type=m6a.xlarge"

bonus

• fixes Autoscaling: provide a timeout before removing a docker node that is "down" #3684, not waiting anymore for modulo of an hour but checking last change on the docker node before switching it off (through EC2_INSTANCES_TIME_BEFORE_TERMINATION)

Related issue/s

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4975 (9d0a3ed) into master (b7f43af) will increase coverage by 0.2%.
The diff coverage is 95.4%.

@@           Coverage Diff            @@
##           master   #4975     +/-   ##
========================================
+ Coverage    87.2%   87.4%   +0.2%     
========================================
  Files        1239    1048    -191     
  Lines       51042   46200   -4842     
  Branches     1080     545    -535     
========================================
- Hits        44528   40424   -4104     
+ Misses       6277    5655    -622     
+ Partials      237     121    -116     

Flag	Coverage Δ
integrationtests	64.9% <100.0%> (-0.1%)	⬇️
unittests	85.0% <95.4%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ckages/models-library/src/models_library/docker.py	89.6% <100.0%> (+0.1%)	⬆️
...ing/src/simcore_service_autoscaling/core/errors.py	100.0% <100.0%> (ø)
...g/src/simcore_service_autoscaling/core/settings.py	100.0% <ø> (ø)
...oscaling/src/simcore_service_autoscaling/models.py	100.0% <100.0%> (ø)
...vice_autoscaling/modules/auto_scaling_mode_base.py	100.0% <100.0%> (ø)
...scaling/modules/auto_scaling_mode_computational.py	100.0% <100.0%> (+1.3%)	⬆️
...e_autoscaling/modules/auto_scaling_mode_dynamic.py	100.0% <100.0%> (ø)
...ing/src/simcore_service_autoscaling/modules/ec2.py	100.0% <100.0%> (ø)
...ore_service_autoscaling/utils/auto_scaling_core.py	94.2% <100.0%> (+3.8%)	⬆️
...service_autoscaling/utils/computational_scaling.py	100.0% <100.0%> (ø)
... and 6 more

... and 197 files with indirect coverage changes

[bisgaard-itis]

Looks good, thanks!

[matusdrobuliak66]

Thanks

[pcrespov]

Very nice job. Left a few comments. thx!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit 9d0a3ed and detected 0 issues on this pull request.

View more on Code Climate.

[GitHK]

👍