Testing upgrade from jupyterhub 3.3.7 to 4.0.0-beta.4

[consideRatio]

I'm using 2i2c infra (2i2c cluster's staging hub) during open source work to check that z2jh 4.0.0-beta.4 is mature enough to be released as 4.0.0. Part of #4970.

The changes made here reflect the kind of changes we may want to roll out systematically at some point. This PR isn't meant to be merged as we don't have a system to install different helm chart versions for different hubs. Instead, I'm just manually deploying this to 2i2c cluster's staging hub to and doing manual checks that for example users home folders remain accessible and servers aren't getting restarted during upgrade.

Issues observed

• JupyterHub startup aborted by failure to connect to an external service (behavior change between jh 4.1.5 and 5.0.0-5.2.0) jupyterhub/jupyterhub#4929
  • Mitigated for 2i2c in 2i2c, staging: fully disable dex #4981, but we also need to handle Including jupyterhub.services.binder.url by default in basehub leads to a warning every minute for non-binder deployments #4939 unless its fixed
  • Fixed by Abort jupyterhub startup only if managed services fail jupyterhub/jupyterhub#4930, released with jupyterhub 5.2.1
  • Included in z2jh via Update jupyterhub from 5.2.0 to 5.2.1 jupyterhub/zero-to-jupyterhub-k8s#3551, released with chart and image version 4.0.0-beta.4.git.6815.hcfafef2a
• (not important) Exiting applicaiton log message to be done at a higher log level than debug jupyterhub/jupyterhub#4928
• Downgrading after upgrade couldn't be done without first replacing the jupyterhub.sqlite file with a backup.
  This can be manually handled by kubectl debug <hub pod name> -it --copy-to=test-debug --image=ubuntu -c hub -- /bin/bash and then looking at /srv/jupyterhub for sqlite backup files and using a backup instead of the one named jupyterhub.sqlite`

What seem to work

Besides the issues, it seems that:

• kubespawner v7
  • nfs storage access was retained as intented by updating subPath: "{username}" to subPath: "{escaped_username}"
  • user servers running while the upgrade was made with JupyterHub 3, 4, and 5 in their images were accessible after jupyterhub was successfully restarted
• oauthenticator v17:
  • cilogon with google as an idp seem to work fine

[consideRatio]

Testing complete, no remaining exceptions to handle for 2i2c deployments besides adjusting subPath config from "{username}" to "{escaped_username}" which I think will be mentioned in z2jh upgrade docs