Node selector support kernels

[amit-chandak-unskript]

Hi @kevin-bates , we want to have dedicated K8s nodes for running EG and kernels. NodeSelector is the way to achieve that, we have modified the deployment for EG to accomplish it, however, how can i pass that nodeSelector when launching the kernel pods? Was looking at launch_kubernetes.py to pass this, but wanted to understand how to make EG pass this nodeSelector map to the launch_kubernetes. Was wondering if there is a way this is already supported, before i start digging into more.

Thanks

[kevin-bates]

Hi @amit-chandak-unskript - thank you for your question.

This kind of customization would likely need to be expressed in the kernel-pod.yaml.j2 script with possible changes to launch_kubernetes.py - which is probably what you were expecting.

The process-proxies currently leave the "provisioning" to whatever is used to launch the pod. Today, this is completely script based, primarily to enable these kinds of customizations.

If there's a way to add this node selector functionality such that, by default, it's not in use or all nodes are available, it would be a much-appreciated contribution that we can document how users can easily achieve this - since this is functionality that would be beneficial to many.

Thanks.

[amit-chandak-unskript]

Thanks @kevin-bates for the quick response. Few follow up questions:

• EG doesnt know whether its running in k8s or not. So, whats a good way to pass this info to EG thats its running inside k8s and has the nodeSelector enabled. Is it via Environment variable or does EG have a config file too?
• I wanted to understand the code flow, i see that launch_kubernetes.py gets called from kernelspecs kernels.json. But want to understand how to pass arguments from EG to this script?
  Thanks in advance

[kevin-bates]

EG does have a config file but for k8s, the configuration is typically driven by environment variables in the helm charts (namely, deployment.yaml.

Any kernelspecs that reference the KubernetesProcessProxy can assume they are being launched from within a k8s cluster.

i see that launch_kubernetes.py gets called from kernelspecs kernels.json. But want to understand how to pass arguments from EG to this script?

That is correct. The argv from the kernel.json is issued within a Popen() call which invokes the launch_kubernetes.py script which, along with the kernel-pod.yaml.j2 jinja template, is responsible for starting the pod. The KubernetesProcessProxy then monitors the pod's startup (and lifecycle) using the kernel's ID as its key to locating the pod.

Currently, there are two ways to influence the kernel pod's configuration.

1. Manually modify each of the appropriate kernel-pod.yaml.j2 files (located in the scripts directory adjacent to the kernel.json file)
2. Convey settings via envs prefixed with KERNEL_ (since those flow to at least the scripts), then apply substitutions within the jinja template (you can see an example of this here for resources and volume mounts.

You will likely be configuring the nodeSelector for the EG deployment in the deployment.yaml. Will the nodeSelector labels (names and values) used for EG be the same for each kernel? If so, it seems like just modifying the helm chart (for EG) and jinja template (for each kernel spec) would be the easiest way forward, bypassing env substitution altogether.

[amit-chandak-unskript]

thanks @kevin-bates. Yes, the nodeselector will be the same as EG and same for each kernel.