JupyterHub users can't create pods on GKE

[mmatiaschek]

My users get this error when trying to start a docker image via JupyterHub.
Error: (422) Reason: error HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b79ba010-af2f-49a6-a606-eda454bb40a5', 'Content-Type': 'application/json', 'Date': 'Fri, 27 Jul 2018 13:45:12 GMT', 'Content-Length': '972'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod "jupyter-accounts-2egoogle-2ecom-3auser12-2ename123-40googlemail-2ecom" is invalid: [spec.volumes[0].name: Invalid value: "volume-accounts-2egoogle-2ecom-3auser12-2ename1234-40googlemail-2ecom": must be no more than 63 characters, spec.containers[0].volumeMounts[0].name: Not found: "volume-accounts-2egoogle-2ecom-3auser12-2ename1234-40googlemail-2ecom"]","reason":"Invalid","details":{"name":"jupyter-accounts-2egoogle-2ecom-3auser12-2ename1234-40googlemail-2ecom","kind":"Pod","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: "volume-accounts-2egoogle-2ecom-3auser12-2ename1234-40googlemail-2ecom": must be no more than 63 characters","field":"spec.volumes[0].name"},{"reason":"FieldValueNotFound","message":"Not found: "volume-accounts-2egoogle-2ecom-3auser12-2ename1234-40googlemail-2ecom"","field":"spec.containers[0].volumeMounts[0].name"}]},"code":422}

related: Is it documented which roles users need to create jupyter pods on gke? Is IAP-Secured Web App User enough?

[jlewi]

This is #1177.

This has been fixed at head but not in 0.2.2. If you copy the contents of
https://github.com/kubeflow/kubeflow/blob/master/kubeflow/core/kubeform_spawner.py

into vendor/kubeflow/core/kubeform_spawner and then rerun

ks apply ${ENV} -c kubeflow-core

That should fix it.

[jlewi]

Duplicate of #1177

[mmatiaschek]

should i try switching to head until 0.3 is released?
unfortunately with the patch i get this error:
Error: (403) Reason: error HTTP response headers: HTTPHeaderDict({'Audit-Id': '32106277-dfbe-4094-a6c5-72a0e1b93ec1', 'Content-Type': 'application/json', 'Date': 'Sat, 28 Jul 2018 22:19:46 GMT', 'Content-Length': '340'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter-mmatiaschek-40gmail-2ecom\" is forbidden: error looking up service account hub/jupyter-notebook: serviceaccount \"jupyter-notebook\" not found","reason":"Forbidden","details":{"name":"jupyter-mmatiaschek-40gmail-2ecom","kind":"pods"},"code":403}

[jlewi]

That might help. I forgot we added a service account to the julyter pods to enable use of TFjob from juypyter.

Deploy.sh at Head will try to setup a Google cloud filestore. This might cause problems for you every if your project is old and has a legacy network. So you might not to comment out those steps in deploy.sh.

[jlewi]

Closing this issue as stale.