How to manage dependencies between different nodes?

[morind]

Hi,
I use JEG based on Spark to provide a multi-tenant/scalable environment for Notebooks based on k8s.
It works pretty well except for Python dependencies.
The problem: some users could install custom dependencies in their notebooks (pip install…) and these dependencies are only installed on Spark Driver node. When Executors are launched, they can be executed on different Pods/Nodes from the Driver and these dependencies are not present.
At the moment we don’t have any distributed filesystem on k8s. If we can avoid that, it would be fine. Does something exist to tackle this problem ?

[kevin-bates]

Thanks for opening this discussion here as well @morind. For completeness, this discussion originated on the Jupyter Community Forum to which I responded with some suggestions. I also suggested David open a discussion here in hopes that folks watching this repo (and not necessarily the forum) could share their solutions to this issue, as I'm sure it's fairly common.

We'd love to hear from you!