S-D26 Machine Learning Support (continuous) Y4M1-12

[KZzizzle]

Dockerized JupyterLabservice with integrated machine learning packages

please, synchronize closely with alessandro and taylor

Items that are already part of ongoing work:

• Resource allocation / scheduler.
• Access to filesystem mount.
• API for running (AI) jobs in the cloud.
• Execution of AI python scripts as "computational services" on data-lake.

Items that are ML-specific and/or new:

• Re-attaching to running services / ability to keep dynamic service alive
• Tensorflow/pytorch-specific solver with GPU capabilities.
• Deployment of tensorflow/pytorch-enabled jupyterlab services.
• Find out whether we need to add a monitor entrypoint to API for interfacing with Tensorboard.
• Creation of a specialized ML computational runner that takes a trained network and data as inputs and gives predictions as output. May or may not require GPU acceleration.
• Creation of example ML jupyter notebooks with skeleton scripts and explanations.
• Support for connections to external databases. May require credentials.
• Capability to publish shareable/reusable models.