personalizable resource limits

[esraneufeld]

To avoid overscheduling of nodes, services should be scheduled with minimal resource requirements. In order not to limit "power users", those requirements can be dynamically be overwritten by users via the osparc user interface. Possibility to override is tied to users/group permissions.
(computational backend for now, also keep in mind #576)

Tasks

Beta Give feedback

1. PublicAPI: it shall be possible to retrieve the solver required resources osparc-simcore#4515
   a:apiserver +1
   
2. PublicAPI: it shall be possible to specify resources when starting a job osparc-simcore#4516
   a:apiserver +1
   
3. webserver shall be able to get the cluster max resources if defined as such osparc-simcore#4357
   a:webserver +1
   

Sundae

Beta Give feedback

1. Migrate projects where override service specifications are necessary osparc-simcore#4468
   2 of 4
   a:infra+ops +1
   

[esraneufeld]

To avoid overscheduling of nodes, services should be scheduled with minimal resource requirements. In order not to limit "power users", those requirements can be dynamically be overwritten by users via the osparc user interface. Possibility to override is tied to users/group permissions.
(computational backend for now, also keep in mind #576)

[sanderegg]

Update on sprint Macarons

Done

• Show services pre-defined resources to user (default resources):
  • ✨ Enhancement/get node resources osparc-simcore#2987
  • Service/Node resources in frontend osparc-simcore#2998

Todo

• allow specific users to change limits on computational resources
  • save resources per node in project
  • handle sharing of projects
• if the resources are not available osparc should not run and show a clear message
• extend to dynamic services

[sanderegg]

Update on sprint Croissant

Done

• Manual assignment of computer nodes for specific users for dynamic-sidecar powered services: ✨ Enhancement/3022/allow setting services scheduler specs (⚠️ devops) osparc-simcore#3034

Ongoing

Open

• Manual overriding services resources (reservations and limits)
• Allow specific users to change limits on computational services in osparc GUI
  • save resources per node in project
  • handle sharing of projects
• if resource not available osparc should not run and show a clear message
• extend to dynamic services

[elisabettai]

@mguidon, could something be done this sprint to prevent the case MS was hitting see this message in Mattermost? For the POs this is still very important and needed for s4l upcoming releases.
Could you please discuss with the backend team and see what can be done?

[mguidon]

Hi. Strictly speaking, personalized resources are not required for s4l:web:lite. Melanies problem is related to her using a very outdated legacy service (with strict resource limits of outrageous 96 GB RAM) and her needing more RAM than the default one for the newer version (16GB).
If we want to work on that I suggest to do a minimal thing that allows us to override the limits but not yet the users. (e.g. enhancing the services_specifications table in the db.)
@sanderegg @GitHK any thoughts on this?

[sanderegg]

Goal for sprint Athena

• allow setting user specific resources on dynamic services (dy-sidecar services only) osparc-simcore#3022

[sanderegg]

Update on sprint Athena

Done

• allow setting user specific resources on dynamic services (dy-sidecar services only) osparc-simcore#3022

[sanderegg]

Goal for sprint Mithril

• Allow service specifications (at least resources such as RAM, CPUs) override for computational services
• deprecate MPI flag

[sanderegg]

Update on sprint Mithril

Done

• ♻️Maintenance: fixes and refactoring osparc-dask-gateway#48
• ⬆️ Removed bokeh dependency + upgrade reqs osparc-dask-gateway#49
• ♻️ Maintenance: cleanup catalog osparc-simcore#3983
• ✨ Personalized resources: Allow override of computational service needed resources (⚠️ devops) osparc-simcore#3989

[sanderegg]

Some notes

current implementation

• modern dynamic services and all computational services can have personalized resource limits/reservations
• these custom changes are set per user/group manually through osparc database
• these changes need access to the said database (dev-team or app-team) and to know what one is doing

If the user need to dynamically change a service resources I see the following requirements:

• UI to allow to change the resource requirements inside a study
• The allowed resource requirements depends on the user/group affiliations and access to actual computational resources (which might be known if running on internal cluster, or unknown if on an external cluster)
• the selected resource requirements shall be saved with the study for persistence
• One might have different default resources based on user/group affiliations
• These defaults will need to be set by some kind of administrator (the cluster's owner)
• When sharing a study with someone else, what happens when these resource requirements are not available to that user/group?

[sanderegg]

Goal for sprint Jelly Beans

Write down in details how this will be surfaced to the users

• Default resources
• Max resoruces
• How to link this to auto-scaling

Potential first use case:
Melanie can choose how much RAM the ti jupyter smash has available. Default will be what is on the label. If the admin allows it, she can individually change the settings upon starting of the service.

[sanderegg]

Update on sprint Jelly Beans

Towards user-defined resources on services plan

1. Running a service with dynamically changed resources

• a service maximum authorized resources per user/groups can be defined (database changes), if not defined would default to the current defaults (defined by service labels or overridden defaults)
• a service defined resources are persisted with the project it is used in and is reloaded with the project
• a service defined resources is visible in the UI
• a service defined resources can be modified within a project up to the max authorized resources for the current user
• a service defined resources is modifiable through the UI
• the backend starts/runs the service based on service within project defined resources
• should be accessible from Public API as well

2. administrator access to a service default resources

• administrator can override the default resources for a service (equivalent to current manual change in the database)
• administrator can set different default resources for different users/groups (equivalent to current manual change in the database
• administrator can change the maximum authorized used resources for a service for different users/groups
• system shall prevent administrator from completely using a shared machine by keeping some factor (e.g. if most powerful machine has 32 CPUs, then the maximum usable could be 8 CPUs)

3. sharing a study

• pre-check if sharee has necessary available resources (similar as user rights)

Use-case:

• Melanie can choose how much RAM the ti jupyter smash has available. Default will be what is on the label. If the admin allows it, she can individually change the settings upon starting of the service.

after discussion with @mguidon the idea is to go with 1. first and wait with 2. until capabilities of dynamic clusters are better defined. 3. could be implemented once 1. is done.

[mguidon]

• Is the administrator one of us or a user? For the latter, how does a user become administrator?
• I think we should default to a selection of cpu/ram and gpu configs that can be chosen from (e.g. high cpu, high ram or similar)

[sanderegg]

• Should also be accessible from the PublicAPI

[sanderegg]

Update for sprint Pastel de Nata

Ongoing

• ✨Personalized limits: add table per service? osparc-simcore#4294

[sanderegg]

Update Watermelon

Done:

• ✨Personalized resource limits: only allow specific groups to override resources osparc-simcore#4417
• Persist resources used by node in project (check comp_tasks, or new table) osparc-simcore#4352
• Frontend shall allow a user to change the service resources if allowed osparc-simcore#4354
• Modify startup of services to validate resources used by service osparc-simcore#4356
• ✨Personalized resource limits: add DB table for max resources per user 🗃️ osparc-simcore#4335
• ✨Personalized resource limits: add project nodes table to store nodes resources 🗃️ osparc-simcore#4362
• ✨Personalized resources: remove projects to projects nodes table 🗃️ osparc-simcore#4384
• ♻️Maintenance: webserver openapi users now auto-generated osparc-simcore#4424
• ♻️Personalized resource limits: remove services limitations table 🗃️ osparc-simcore#4428
• Personalized resource limits: frontend to allow users to change limits based on user permissions osparc-simcore#4435
• Personalized resource limits: frontend to handle error osparc-simcore#4436
• 🐛Personalized-resource-limits: return default value when no product is defined osparc-simcore#4449

Summary:
The user can now change a service required resources (CPUs, RAM) through the oSparc GUI. These changes are persisted within the project. There are currently no upper bound for resources. The platform will try to start the service using the defined required resources.

Open:

• PublicAPI changes to present the same options as the GUI
• Allow to select pre-defined EC2 instances to run services

[sanderegg]

Update Sundae

Ongoing

• Migrate projects where override service specifications are necessary osparc-simcore#4468