Update to z2jh 2 and JupyterHub 3 #1544

consideRatio · 2022-10-10T14:06:22Z

This better be done at some point in time, and one reason to do it is because it makes us compatible with k8s 1.25 and higher among other things.

Closes CI system broken with k8s 1.25 (at least k3s 1.25) #1541
Enables resolving Support ...image.pull[Policy|Secrets] consistently and hook into JupyterHub's chart wide configuration for imagePullSecrets #1444 in a way like z2jh has done it

Since this may be relevant knowing about I pinged a lot of you who I guessed could be influenced.

Draft status of PR because of ...

[D 2022-10-10 14:15:14.238 JupyterHub app:2035] Loading roles into database
[I 2022-10-10 14:15:14.249 JupyterHub roles:173] Role jupyterhub-idle-culler added to database
[I 2022-10-10 14:15:14.251 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[W 2022-10-10 14:15:14.264 JupyterHub app:2312] Service binder sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2022-10-10 14:15:14.265 JupyterHub roles:238] Adding role admin for Service: binder
[E 2022-10-10 14:15:14.268 JupyterHub app:3297]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3294, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2824, in initialize
        self.init_services()
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2338, in init_services
        setattr(service, key, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 715, in __set__
        self.set(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 689, in set
        new_value = self._validate(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 723, in _validate
        value = self._cross_validate(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 729, in _cross_validate
        value = obj._trait_validators[self.name](obj, proposal)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 1132, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/services/service.py", line 322, in _validate_client_id
        raise ValueError(
    ValueError: service  has oauth_client_id='binder-oauth-client-test'. Service oauth client ids must start with 'service-'
    
[D 2022-10-10 14:15:14.270 JupyterHub application:961] Exiting application: jupyterhub

And failures following trialing prefixing with service-. Feel free to push commits to resolve this!

consideRatio · 2022-10-12T14:42:25Z

JupyterHub logs of high relevance

[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

Additional log lines for more context

[W 2022-10-10 14:34:45.684 JupyterHub app:2312] Service binder sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2022-10-10 14:34:45.685 JupyterHub roles:238] Adding role admin for Service: binder
[I 2022-10-10 14:34:45.691 JupyterHub provider:651] Creating oauth client service-binder
[I 2022-10-10 14:34:45.706 JupyterHub app:2244] Adding API token for service: binder
[D 2022-10-10 14:34:45.712 JupyterHub app:2274] Purging expired APITokens
[D 2022-10-10 14:34:45.713 JupyterHub app:2274] Purging expired OAuthCodes
[D 2022-10-10 14:34:45.716 JupyterHub app:2110] Loading role assignments from config
[W 2022-10-10 14:34:45.729 JupyterHub app:2657] Allowing service binder to complete OAuth without confirmation on an authorization web page
...
[I 2022-10-10 14:34:49.605 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (@::ffff:10.1.1.85) 1.39ms
[I 2022-10-10 14:34:49.650 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (@::ffff:10.1.1.85) 33.09ms
[D 2022-10-10 14:34:49.662 JupyterHub roles:281] Assigning default role to User dummy
[I 2022-10-10 14:34:49.667 JupyterHub roles:238] Adding role user for User: dummy
[D 2022-10-10 14:34:49.672 JupyterHub roles:281] Assigning default role to User dummy
[D 2022-10-10 14:34:49.676 JupyterHub base:559] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2022-10-10 14:34:49.676 JupyterHub base:563] Setting cookie for dummy: jupyterhub-hub-login
[D 2022-10-10 14:34:49.676 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-10-10 14:34:49.677 JupyterHub base:810] User logged in: dummy
[I 2022-10-10 14:34:49.677 JupyterHub log:186] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.1.85) 18.70ms
[D 2022-10-10 14:34:49.690 JupyterHub base:275] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-10 14:34:49.694 JupyterHub provider:415] Validating client id service-binder
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri http://127.0.0.1:8585/oauth_callback for client service-binder.
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.700 JupyterHub provider:490] validate_redirect_uri: client_id=service-binder, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.701 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes ['access:services!service=binder', 'read:users:name!user', 'read:users:groups!user'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2022-10-10 14:34:49.702 JupyterHub provider:614] Allowing request for scope(s) for service-binder:  read:users:groups!user,access:services!service=binder,read:users:name!user
[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

Origin of error message

https://github.com/jupyterhub/jupyterhub/blob/5997614f452a3cfebfa541800012a914d37f3ec8/jupyterhub/apihandlers/auth.py#L265-L283

rbac.enabled was renamed to rbac.create in z2jh 2.0.0

This reverts commit a515e23.

consideRatio · 2022-10-12T23:30:48Z

I think I've pinpointed the differences of relevance here.

The different outcome is determined in the GET webrequest handler for JupyterHub's /api/oauth2/authorize path, specifically this code segment from jupyterhub 3.0.0 and this code segment from jupyterhub 1.5.0.

In 3.0.0 we get User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder, while in 1.5.0 we get Skipping oauth confirmation for <User(dummy 0/1 running)> accessing JupyterHub service binder. The JupyterHub registered service binder is configured with oauth_no_confirm: true, and that is why we observe the "Skipping oauth confirmation" message. Should we do that also for 3.0.0? If so, that logic maybe needs to be given higher priority in JupyterHub.

With z2jh 2.0.0 and JupyterHub 3.0.0

[I 2022-10-10 14:34:49.677 JupyterHub base:810] User logged in: dummy
[I 2022-10-10 14:34:49.677 JupyterHub log:186] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.1.85) 18.70ms
[D 2022-10-10 14:34:49.690 JupyterHub base:275] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-10 14:34:49.694 JupyterHub provider:415] Validating client id service-binder
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri http://127.0.0.1:8585/oauth_callback for client service-binder.
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.700 JupyterHub provider:490] validate_redirect_uri: client_id=service-binder, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.701 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes ['access:services!service=binder', 'read:users:name!user', 'read:users:groups!user'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2022-10-10 14:34:49.702 JupyterHub provider:614] Allowing request for scope(s) for service-binder:  read:users:groups!user,access:services!service=binder,read:users:name!user
[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

With z2jh 1.2.0 and JupyterHub 1.5.0

[I 2022-10-12 10:31:27.419 JupyterHub base:762] User logged in: dummy
[I 2022-10-12 10:31:27.419 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dbinder-oauth-client-test%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjZlYmM2YzgyZDk1ODQ2MjNhMmU4ZGNhOWFjYjJhNTg2IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=binder-oauth-client-test&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.0.155) 16.90ms
[D 2022-10-12 10:31:27.429 JupyterHub base:283] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-12 10:31:27.431 JupyterHub provider:409] Validating client id binder-oauth-client-test
[D 2022-10-12 10:31:27.437 JupyterHub provider:484] validate_redirect_uri: client_id=binder-oauth-client-test, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-12 10:31:27.439 JupyterHub auth:252] Skipping oauth confirmation for <User(dummy 0/1 running)> accessing JupyterHub service binder
[D 2022-10-12 10:31:27.439 JupyterHub provider:409] Validating client id binder-oauth-client-test
[D 2022-10-12 10:31:27.440 JupyterHub provider:484] validate_redirect_uri: client_id=binder-oauth-client-test, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-12 10:31:27.440 JupyterHub provider:231] Saving authorization code binder-oauth-client-test, uPo..., (), {}
[I 2022-10-12 10:31:27.444 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=binder-oauth-client-test&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] -> http://127.0.0.1:8585/oauth_callback?code=[secret]&state=[secret] (dummy@::ffff:10.1.0.155) 15.75ms

not users

consideRatio · 2022-10-13T11:57:01Z

🎉 wiee nice @minrk I see the tests passing with your changes!

Sorry for missing to bump the jupyterhub version, we even had a note about it... DOH!

minrk · 2022-10-13T11:59:33Z

I believe the missing bit is that users must be granted access to services explicitly via load_roles.

That's the

User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder

message, which translates to "user dummy does not have scope access:services!binder." Granting users access to services via loadRoles should get past this step, at least. I added access:services to the user role.

JupyterHub in the binderhub image also needs to be updated to match, which I added as well.

The binder service also should not need to be admin, we should define Binder's own permissions in a role. It needs:

servers to read, start, and stop user servers
admin:users (if not using auth)

It may make sense to define the role(s) dynamically in Python, because of the if auth_enabled logic involved.
Or we could define two roles statically in config, and then assign the role dynamically, depending on if auth is enabled or not.

Alternately, it could get no permissions at all in the auth case, and we could perform the launch requests with the user's own token. That might be better! It would mean specifying the permission servers!user in the services.binder.oauth_client_allowed_scopes config. It would still mean defining the (default) binder role (or not) and then assigning it (or not) based on auth_enabled.

consideRatio · 2022-10-13T13:30:33Z

@minrk I opened #1553 to represent your comments suggestions on how to tighten the requested permissions needed.

Thanks for completing this PR, this LGTM!

jupyterhub/binderhub#1544 Merge pull request #1544 from consideRatio/pr/update-z2jh

manics · 2022-10-14T11:30:04Z

Do we need to update repo2docker to match the JupyterHub bump?
https://github.com/jupyterhub/repo2docker/blob/d7be04efb590a2a1227614ccaf9b504ad9333470/repo2docker/buildpacks/conda/environment.yml#L11

consideRatio · 2022-10-14T12:55:03Z

Yepp, we need a new jupyterhub version for compatibility i think :/

manics · 2022-10-14T19:38:26Z

I think we should make one last release before bumping it: jupyterhub/repo2docker#1194

consideRatio requested review from yuvipanda, betatim, GeorgianaElena, manics, minrk, sgibson91, callummole and mael-le-gal October 10, 2022 14:15

consideRatio added the maintenance Under the hood improvements and fixes label Oct 10, 2022

consideRatio mentioned this pull request Oct 12, 2022

ci: extend test timeout #1549

Closed

minrk added the breaking label Oct 12, 2022

consideRatio mentioned this pull request Oct 12, 2022

ci - do not merge: debug failure #1542

Closed

consideRatio force-pushed the pr/update-z2jh branch from 90246b3 to b09d282 Compare October 12, 2022 19:55

consideRatio added 5 commits October 12, 2022 21:57

Update to z2jh 2.0.0

ee968d7

Rely on default set for rbac.enabled/create

278c27a

rbac.enabled was renamed to rbac.create in z2jh 2.0.0

Omit configuration of oauth_client_id, rely on its default value

6386007

Set JUPYTERHUB_SERVICE_NAME

31676a7

Revert "ci: test k3s 1.24"

f838b92

This reverts commit a515e23.

consideRatio force-pushed the pr/update-z2jh branch from b09d282 to f838b92 Compare October 12, 2022 19:58

consideRatio added 3 commits October 12, 2022 22:08

trial and error setting env var

8224ab3

trial and error setting env var

74c5f81

trial and error - using HubOAuth.access_scopes

79f94ce

minrk force-pushed the pr/update-z2jh branch from 937feb2 to f6ee4e8 Compare October 13, 2022 11:22

auth test: grant users access to binder services

c79e627

minrk force-pushed the pr/update-z2jh branch from f6ee4e8 to c79e627 Compare October 13, 2022 11:26

minrk added 3 commits October 13, 2022 13:36

upgrade jupyterhub to 3.* in binderhub image

543fcd7

binder doesn't need to be admin in jupyterhub 2.0

39980f0

user role is called user

30bdb62

not users

consideRatio changed the title ~~Update to z2jh 2.0.0~~ Update to z2jh 2 and JupyterHub 3 Oct 13, 2022

minrk approved these changes Oct 13, 2022

View reviewed changes

consideRatio mentioned this pull request Oct 13, 2022

Reduce requested JupyterHub permissions and switch to using a role #1553

Open

consideRatio merged commit d8f0e3b into jupyterhub:master Oct 13, 2022

consideRatio pushed a commit to jupyterhub/helm-chart that referenced this pull request Oct 13, 2022

[binderhub] Automatic update for commit 0.2.0-0.dev.git.2789.h39980f0

38469ec

jupyterhub/binderhub#1544 Merge pull request #1544 from consideRatio/pr/update-z2jh

manics mentioned this pull request Oct 14, 2022

Initial changelog for 2022.10.0 jupyterhub/repo2docker#1194

Merged

1 task

manics mentioned this pull request Oct 14, 2022

Supporting JupyterHub singleuser 3.0.0 jupyterhub/repo2docker#1195

Closed

This was referenced Oct 19, 2022

Updates binderhub chart to 1.0.0-0.dev.git.2797.h951750e jupyterhub/mybinder.org-deploy#2367

Closed

Updates binderhub chart to 1.0.0-0.dev.git.2816.h5db2f98 jupyterhub/mybinder.org-deploy#2376

Merged

manics mentioned this pull request May 28, 2024

Question about docs for auth - what makes jupyterhub-singleuser cmd change required? #1854

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to z2jh 2 and JupyterHub 3 #1544

Update to z2jh 2 and JupyterHub 3 #1544

consideRatio commented Oct 10, 2022 •

edited

Loading

consideRatio commented Oct 12, 2022 •

edited

Loading

consideRatio commented Oct 12, 2022

consideRatio commented Oct 13, 2022

minrk commented Oct 13, 2022 •

edited

Loading

consideRatio commented Oct 13, 2022 •

edited

Loading

manics commented Oct 14, 2022

consideRatio commented Oct 14, 2022

manics commented Oct 14, 2022

Update to z2jh 2 and JupyterHub 3 #1544

Update to z2jh 2 and JupyterHub 3 #1544

Conversation

consideRatio commented Oct 10, 2022 • edited Loading

Draft status of PR because of ...

consideRatio commented Oct 12, 2022 • edited Loading

JupyterHub logs of high relevance

Additional log lines for more context

Origin of error message

consideRatio commented Oct 12, 2022

With z2jh 2.0.0 and JupyterHub 3.0.0

With z2jh 1.2.0 and JupyterHub 1.5.0

consideRatio commented Oct 13, 2022

minrk commented Oct 13, 2022 • edited Loading

consideRatio commented Oct 13, 2022 • edited Loading

manics commented Oct 14, 2022

consideRatio commented Oct 14, 2022

manics commented Oct 14, 2022

consideRatio commented Oct 10, 2022 •

edited

Loading

consideRatio commented Oct 12, 2022 •

edited

Loading

minrk commented Oct 13, 2022 •

edited

Loading

consideRatio commented Oct 13, 2022 •

edited

Loading