Install JupyterHub in the cern-vre cluster via tf helm provider

[goseind]

Information

• Versions of the helm chart: https://jupyterhub.github.io/helm-chart/
• Chart: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/Chart.yaml and configuration ref: https://z2jh.jupyter.org/en/stable/resources/reference.html#helm-chart-configuration-reference
• Docs: https://github.com/jupyterhub/helm-chart
• https://github.com/jupyterhub/repo2docker
• OAuth configuration and examples: https://z2jh.jupyter.org/en/stable/administrator/authentication.html#oauth2-based-authentication

Questions

• (how) can sessions be stored for longer periods of time? e. g. https://pypi.org/project/dill/
• would a similar solution to Codespaces/Gitpod self-hosted (https://www.gitpod.io/docs/configure/self-hosted/latest) be a solution?
• compare to SWAN and Binderhub: https://jupyter.org/binder solutions

Tasks

Beta Give feedback

1. Install z2jh helm chart with basic configuration
2. Expose the service to the outside with SSL
3. Configure auth/login with IAM ESCAPE VO instance
4. Create a new database called jhub in vre dbod instance https://dbod.web.cern.ch/pages/dashboard
5. Install RUCIO plugin #53
   +0
6. Transfer and rebuild images for singleuser JupyterHub profiles #45
   2 of 2
   cern-vre-infra component/images component/jhub +3
   
7. Open the Firewall for this service
8. Configure CVMFS as an extraVolume
9. Configure additional extra volumes
10. Transfer other relevant settings from the old VRE configuration (reducing complexity)
11. Solve problem/configure user persistent storage with PVCs
12. Think about other useful improvements and configuration settings for the chart
13. Documentation on everything in the Wiki
14. Investigate https://github.com/jupyterhub/binderhub installation
15. Investigate multiple sessions ref. Support multiple open JupyterLab windows: multiple state databases. jupyterlab/jupyterlab#4041
16. changing rse S3 secret #64
    +0
17. Move chart to flux

[goseind]

Why did the ESCAPE team decide to go with singleuser.storage.static instead of dynamically provisioned PVCs through a storage class and how does Jhub provide individual storage with a static PVC?

• ESCAPE JHub values: https://gitlab.cern.ch/escape-wp2/flux-rucio/-/blob/master/escape/jupyterhub/releases/jupyterhub.yaml
• JHub Storage config options: https://z2jh.jupyter.org/en/stable/jupyterhub/customizing/user-storage.html
• File Share CERN docs: https://clouddocs.web.cern.ch/file_shares/quickstart.html

tag @garciagenrique

[garciagenrique]

• JupyterHub was installed following the zero2jupyterhub documentation and using helm charts.

  • The config.yaml file is empty so that z2jh starts with the default values.
  • However, the deployment won't succeed because there is no storage assigned to the cluster. See below:

• Currently the HUB pod is starting (Pending mode) with the following error

|   Type     Reason            Age                  From               Message                                    │
│   ----     ------            ----                 ----               -------                                    │
│   Warning  FailedScheduling  19m (x309 over 26h)  default-scheduler  0/3 nodes are available: 3 pod has unbound │
│  immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for schedul │
│ ing.

This error is connected with the problem @goseind is stressing above: why static vs dynamic storage for the cluster.

[garciagenrique]

The following commands where used for the deployment (helm needs to be installed, of course)

$ helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
$ helm repo update

To install/updated, just modify the config.yaml file, save it and run

$ helm upgrade --cleanup-on-fail \
  --install z2jh jupyterhub/jupyterhub \
  --namespace jupyterhub \
  --create-namespace \
  --version=2.0.0 \
  --values config.yaml

The des installation of the helm chart does not work really well. The way it's currently done is by logging into k9s, selecting the jupyterhub (or whoever it was called), and erasing it completely.

[goseind]

I have requested a quota change for the number of shares, so we can use dynamically provisioned storage.

[goseind]

Done, quota updated to 200 shares.

[goseind]

JupyterHub is running under http://137.138.226.35 (IP subject to change). Current status:

• Update on Quota Question here
• LBaaS automatic LB provisioning was having issues again (see SNOW issue), thus creating the LB manually like so openstack loadbalancer create --name kube_service_8d85602e-7816-4153-b85b-ea86586e47c1_jhub_proxy-public --vip-network-id public and passing the IP into the helm values loadBalancerIP.
• sqlite-memory db type is just for testing and needs to be replaced by DBOD postgres db.
• Due to the usage of podSecurityContext also here, we need to set fsGroupChangePolicy: "OnRootMismatch" as per default the shares are owned by root:root, and writable only by root (ref. https://kubernetes.docs.cern.ch/docs/storage/fileshares/#running-pods-with-fsgroup-security-context). Setting the value through extraPodConfig as described here, does not work, the value does not get passed onto the users container. Topic opened in Forum: https://discourse.jupyter.org/t/extrapodconfig-values-do-not-get-set/17868 (see also Mattermost)

tf commands:

terraform apply -target=kubernetes_namespace_v1.ns_jupyterhub
terraform apply -target=kubernetes_storage_class_v1.sc_manila-meyrin-cephfs # sc with Delete policy did not exist yet
terraform destroy -target=helm_release.jupyterhub-chart

@garciagenrique see if you can follow my changes, the extra values I set are described in the customization docs. As I have no idea why the value does not get set, I'll have to wait for an answer in the Forum..

[goseind]

My workaround solution by setting the fsGroup to 0 in order to use root access didn't work as then the following error occurs: Running as root is not recommended. Use --allow-root to bypass. This has been a problem for other users too, see: jupyterhub/zero-to-jupyterhub-k8s#562 or jupyterhub/zero-to-jupyterhub-k8s#2177

Also asking the SWAN team how they worked around this ref.: https://github.com/swan-cern/swan-charts/blob/master/swan/values.yaml#L22

Another idea would be to use extraConfig to directly modify KubeSpawner.

• Delete PVCs used fo testing

[goseind]

To solve the issue with the extraPodConfig not being set, I created a bug report in the JupyterHub repo: jupyterhub/zero-to-jupyterhub-k8s#3021

[garciagenrique]

@goseind could we just not set up a podSecurityContext for the moment, finish configuring JHub, and later implement this ?

[goseind]

Set a meeting with Diogo from SWAN for next week Monday and check their storage configuration with eosxd: https://gitlab.cern.ch/kubernetes/automation/charts/cern/-/tree/master/eosxd

[goseind]

For an update see #34, the service is now reachable from within CERN but needs further configuration, as listed in the issue description.
@garciagenrique can you add a PR with the cvmfs configuration?

[garciagenrique]

The base image of the ESCAPE-VRE is this one: https://gitlab.cern.ch/escape-wp2/docker-images/-/tree/master/datalake-singleuser
There are some features that could be improved (maybe python version and latests rucio client versions ?), but all the configuration of how to add the rucio-jupyterlab plugin is linked here.

I suggest either start with this image, either based on it.

[wiegerthefarmer]

@goseind could we just not set up a podSecurityContext for the moment, finish configuring JHub, and later implement this ?

is there a solution to this? I've tried everything to get fsGroupChangePolicy: "OnRootMismatch" set. Nothing gets passed to the pod. Only setting the values in a yaml that is used to start the pod works. But nothing in the spawner works.

[goseind]

@goseind could we just not set up a podSecurityContext for the moment, finish configuring JHub, and later implement this ?

is there a solution to this? I've tried everything to get fsGroupChangePolicy: "OnRootMismatch" set. Nothing gets passed to the pod. Only setting the values in a yaml that is used to start the pod works. But nothing in the spawner works.

So far we haven't found a solution either, some values get set through the YAML values, but for the ones we need not, setting them with the extra config script doesn't work either. We need to further debug this.

[goseind]

• Adjust the K8s network policy of JHub for URLs to work (optional?)
• use Lets Encrypt cert with cert-manager
• Change single-user storage to share once the fsGroup issue is solved
• rebuild main single-user images on GH (see my example repo)
• Move deployment to ArgoCD once set up
• Redo EOS FUSE mount with CERN-provided image or rebuild the old one here (use private image as the keytab needs to stay secret), ref. to: Enable EOS mount OpenStack #130

[goseind]

@garciagenrique I now merged the config, from my side this looks fine. Do you want to take charge of redoing the images, also the EOS image? References can be found in the config of the images I was using/creating.

[goseind]

I think this issue could be closed and the remaining tasks split up into separate tasks as they are not directly linked to the initial goal of this issue. What do you think @garciagenrique ?

[garciagenrique]

I agree @goseind, although this issue become too big and touched plenty of subjects...

Could we start in this thread an interaction to summary the remaining tasks ?

[goseind]

In my opinion, this topic can now be closed as all the tasks have been completed.