Computational infrastructure

[mrocklin]

During the sprint I expect that people will work most of the time on their laptops. However it might also be useful to have some distributed services available, either on the cloud with a JupyterHub-Kubernetes deployment or on an HPC system. Is this something that we want to manage centrally or should we expect people to have their own preferred computational systems?

[NelleV]

I can probably try to have something set up here at UC Berkeley, based on jupyterhub or the xsede computing services.

[NelleV]

@yuvipanda happens to have a group of machines waiting for a group of users to use them.

Yuvi, do you mind describing what you could maybe provide?

[yuvipanda]

The Pacific Research Platform has a Kubernetes Cluster they're eager to put to test with real world use cases. This gives you a namespace in a Kubernetes cluster with CPU, Memory, disk and some GPUs. I already have a JupyterHub running there, and I'm also experimenting with providing SSH Based access(check it out with ssh -t <github-username>@kubessh.nautilus.optiputer.net -p 32222 -- --image=jupyter/base-notebook, with your github ssh key & any docker image).

If y'all can give me more details on what kinda workloads you'd likely be running I can say more :)

[mrocklin]

I would default to something Pangeo-like. People will want the common set of analytics libraries like scikit-learn, scikit-image, numpy, pandas, etc.. They'll also want to use something like dask_kubernetes to scale these out on the cluster. This scaling will likely be very bursty, with use somewhere between minutes and hours. I expect that most distributed deployments will use something like 50 cores, though we will likely want to expand out to 1000 or so if available.

…

On Thu, May 3, 2018 at 5:51 PM, Yuvi Panda ***@***.***> wrote: The Pacific Research Platform has a Kubernetes Cluster they're eager to put to test with real world use cases. This gives you a namespace in a Kubernetes cluster with CPU, Memory, disk and some GPUs. I already have a JupyterHub running there, and I'm also experimenting with providing SSH Based access(check it out with ssh -t ***@***.*** nautilus.optiputer.net -p 32222 -- --image=jupyter/base-notebook, with your github ssh key & any docker image). If y'all can give me more details on what kinda workloads you'd likely be running I can say more :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBlEsPfzsZ-eNMyEExJqWhGLhRt7ks5tu3vSgaJpZM4TrxZV> .