✨ Personalized resource limits: add API to change node resources ⚠️🗃️

[sanderegg]

What do these changes do?

• open webserver API
• currently allows changes with no limits !!! --> reserved to TESTER users only for now then we will add per group allowance
• webserver API for getting/putting node resources now directly connects with projects_nodes table @odeimaiz please ensure you do not use catalog//resources as this returns the default values not the ones saved into the project. The projects//nodes/*/resources also returns the default if it's missing from the database. Or let's check if this has any need. plan is to deprecate it.
• udpated project duplication procedure to also copy project nodes table with defined resources
• updated director-v2 computations creation to also use the projects_nodes table instead of direct call to catalog
• updated dynamic services startup procedure to take the project node resources instead of default
• default uses the service docker labels or whatever might be changed in services_specifications table as before
• database migration code to create all the necessary projects_nodes entries required by each project (this may take some time to migrate, local tests with staging dalco takes about 10-20 seconds) ⚠️🗃️
• all pytest fixtures in webserver/director-v2/postgres-database are adapted to also create entries in projects_nodes

Notice the procedure now for resources:

• When a project is created, and nodes added currently both the projects/*workbench and a new row in projects_nodes is added with the default resources as given by the catalog service
• If this is a dynamic node, at start the webserver retrieves the resources and uses them to start the service
• If this is a computational node, at start the director-v2 retrieves the resources and uses them to schedule the service on the dask backend.

Bonuses:

• utils_projects_nodes now allows bulk insertion instead of sequential, much faster
• IMPORTANT NOTE: testing postgres-database will now turn into errors anything related to pre-sql 2.0 code
• removed warnings that says DO NOT USE IN PRODUCTION, STILL UNDER DEVELOPMENT as it's a lie
• skip test_acquire_all_available_node_locks_stress_test @GitHK to be removed with remove node rights from director-v2 #4405
• test_project_db.py got some magnitude of orders faster

Related issue/s

• fixes Persist resources used by node in project (check comp_tasks, or new table) #4352

How to test

make build up-prod
# then head to create a study, add a service
# go to check the projects_nodes table in adminer
# change the values there, or use the /dev/doc entrypoint
# break your computer 💥 

DevOps Checklist

• breaks availability if used wildly. but not sure that is worse than before. reserved for TESTER users for now.

[codecov]

Codecov Report

Merging #4374 (0f80be1) into master (7047efc) will decrease coverage by 0.5%.
The diff coverage is 96.6%.

@@           Coverage Diff            @@
##           master   #4374     +/-   ##
========================================
- Coverage    86.1%   85.7%   -0.5%     
========================================
  Files         954     995     +41     
  Lines       40291   42585   +2294     
  Branches     1007    1009      +2     
========================================
+ Hits        34718   36521   +1803     
- Misses       5342    5832    +490     
- Partials      231     232      +1     

Flag	Coverage Δ
integrationtests	54.6% <49.6%> (-13.4%)	⬇️
unittests	83.7% <96.6%> (+0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rc/simcore_service_webserver/projects/_db_utils.py	97.1% <ø> (ø)
...imcore_service_webserver/security/_access_roles.py	100.0% <ø> (ø)
..._service_webserver/studies_dispatcher/_projects.py	93.8% <ø> (ø)
...ce_webserver/studies_dispatcher/_studies_access.py	91.1% <ø> (ø)
...simcore_service_webserver/projects/projects_api.py	93.1% <88.0%> (-0.4%)	⬇️
.../simcore_postgres_database/utils_projects_nodes.py	96.0% <90.6%> (-4.0%)	⬇️
..._director_v2/modules/db/repositories/comp_tasks.py	97.3% <95.2%> (+0.1%)	⬆️
...s-library/src/models_library/services_resources.py	90.7% <100.0%> (ø)
...base/src/simcore_postgres_database/utils_models.py	100.0% <100.0%> (ø)
...rary/src/servicelib/aiohttp/requests_validation.py	86.3% <100.0%> (ø)
... and 7 more

... and 63 files with indirect coverage changes

[codeclimate]

Code Climate has analyzed commit 0f80be1 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[matusdrobuliak66]

Reviewed 👍 really nice