Re-attach to running services / keeping dynamic services alive

[newton1985]

Describe the user role
As an oSPARC user, I would like to be able to re-attach to running interactive services.

Describe the goal
Currently, logging out of a project or even exiting from a service back to the project window kills interactive services. It would be nice to be able to keep them alive and re-attach.

Describe the benefit
This would greatly enhance the development of AI pipelines, by allowing for parallel work, letting computations run in the background, and/or preserving the state of a kernel so as not to lose variables of interest.

Additional context
Refers to #332

[pcrespov]

Some tips on the implementation

• When a project closes, all dy-services in it are closed by the GC
  • flag services to avoid GC
• If I have all projects closed, how can I know which/what is running?
  • BE: API to get which services in a closed project are still running
  • UI: projects in dashboard now can show that are still open
  • UI/BE: Extra resource usage page with table and columns "project, service, RAM, storage, cpu, etc"
• Review policy to start/stop dynamic-services.
  • Right now when a project opens, all dy-services are started (this is sometimes wasting resources e.g. when user just want to layout the pipeline).
  • When it closes, all dy-services are stopped, but we want to give the option to the user to avoid this default behaviour

[sanderegg]

Currently the resources used by dynamic services are clearly defined, and they are removed once the service closes.
I think this raises a few questions:

• Letting a service up means the resources it is using are not available for others. Who/How do we define such rights? How many such services are allowed? do we allow this on the common cluster? or only on dedicated ones?
• Forgetting that there is a service running forever will happen, what do we do for such cases? a time limit? Trying to detect inactivity? Shall we force a user to log in from time to time to be sure ?
• Computational services work as you describe. They continue running in the background. Maybe that could be a solution for such tasks?