Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop forcing use of NotebookApp, take 2 #2210

Merged
merged 2 commits into from
Feb 16, 2023

Conversation

yuvipanda
Copy link
Member

@yuvipanda yuvipanda commented Feb 15, 2023

Reverts #2203

We had to revert it because nbgitpuller on cloudbank hubs failed when we started using jupyter
server everywhere - I am not entirely sure why, as it had worked before! We then upgraded just the
version of jupyterhub and nbgitpuller (2i2c-org/2i2c-hubs-image#3), then tried
to do so without causing massive upgrades in middle of semester for all other packages (2i2c-org/2i2c-hubs-image#4).
However, the image produced fails to even start now, with:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 5, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
    from .app import SingleUserNotebookApp, main
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser/app.py", line 15, in <module>
    from .mixins import make_singleuser_app
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/singleuser/mixins.py", line 44, in <module>
    from ..log import log_request
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/log.py", line 14, in <module>
    from .handlers.pages import HealthCheckHandler
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/handlers/__init__.py", line 1, in <module>
    from . import base, login, metrics, pages
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 24, in <module>
    from .. import __version__, orm, roles, scopes
  File "/opt/conda/lib/python3.7/site-packages/jupyterhub/orm.py", line 31, in <module>
    from sqlalchemy.orm import (
  File "/opt/conda/lib/python3.7/site-packages/sqlalchemy/orm/__init__.py", line 56, in <module>
    from .decl_api import add_mapped_attribute as add_mapped_attribute
  File "/opt/conda/lib/python3.7/site-packages/sqlalchemy/orm/decl_api.py", line 164, in <module>
    query_expression,
TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'

in the user pod logs.

2i2c-org/2i2c-hubs-image#5 is an attempt to fix this. Note that this is
the default image for all our 2i2c hubs as well, not just cloudbank!

We need this PR to go in to support latest pangeo docker images and latest jupyterlab, but
it can't until we have upgraded our other images.

@github-actions
Copy link

github-actions bot commented Feb 15, 2023

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider Cluster Name Upgrade Support? Reason for Support Redeploy Upgrade Staging? Reason for Staging Redeploy
aws carbonplan No Yes Core infrastructure has been modified
aws gridsst No Yes Core infrastructure has been modified
gcp m2lines No Yes Core infrastructure has been modified
gcp awi-ciroh No Yes Core infrastructure has been modified
gcp callysto No Yes Core infrastructure has been modified
gcp pangeo-hubs No Yes Core infrastructure has been modified
aws ubc-eoas No Yes Core infrastructure has been modified
gcp 2i2c No Yes Core infrastructure has been modified
aws openscapes No Yes Core infrastructure has been modified
aws nasa-veda No Yes Core infrastructure has been modified
aws uwhackweeks No Yes Core infrastructure has been modified
gcp linked-earth No Yes Core infrastructure has been modified
gcp meom-ige No Yes Core infrastructure has been modified
gcp leap No Yes Core infrastructure has been modified
kubeconfig utoronto No Yes Core infrastructure has been modified
gcp cloudbank No Yes Core infrastructure has been modified
aws 2i2c-aws-us No Yes Core infrastructure has been modified
aws victor No Yes Core infrastructure has been modified
aws nasa-cryo No Yes Core infrastructure has been modified
gcp 2i2c-uk No Yes Core infrastructure has been modified

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
aws carbonplan prod Core infrastructure has been modified
aws gridsst prod Core infrastructure has been modified
gcp m2lines prod Core infrastructure has been modified
gcp awi-ciroh prod Core infrastructure has been modified
gcp callysto prod Core infrastructure has been modified
gcp pangeo-hubs prod Core infrastructure has been modified
aws ubc-eoas prod Core infrastructure has been modified
gcp 2i2c demo Core infrastructure has been modified
gcp 2i2c ohw Core infrastructure has been modified
gcp 2i2c pfw Core infrastructure has been modified
gcp 2i2c peddie Core infrastructure has been modified
gcp 2i2c catalyst-cooperative Core infrastructure has been modified
gcp 2i2c earthlab Core infrastructure has been modified
gcp 2i2c paleohack2021 Core infrastructure has been modified
gcp 2i2c aup Core infrastructure has been modified
gcp 2i2c temple Core infrastructure has been modified
gcp 2i2c ucmerced Core infrastructure has been modified
aws openscapes prod Core infrastructure has been modified
aws nasa-veda prod Core infrastructure has been modified
gcp linked-earth prod Core infrastructure has been modified
gcp meom-ige prod Core infrastructure has been modified
gcp meom-ige drakkar-demo Core infrastructure has been modified
gcp leap prod Core infrastructure has been modified
kubeconfig utoronto prod Core infrastructure has been modified
kubeconfig utoronto r-prod Core infrastructure has been modified
gcp cloudbank ccsf Core infrastructure has been modified
gcp cloudbank csm Core infrastructure has been modified
gcp cloudbank elcamino Core infrastructure has been modified
gcp cloudbank glendale Core infrastructure has been modified
gcp cloudbank howard Core infrastructure has been modified
gcp cloudbank miracosta Core infrastructure has been modified
gcp cloudbank skyline Core infrastructure has been modified
gcp cloudbank demo Core infrastructure has been modified
gcp cloudbank fresno Core infrastructure has been modified
gcp cloudbank lassen Core infrastructure has been modified
gcp cloudbank sbcc Core infrastructure has been modified
gcp cloudbank lacc Core infrastructure has been modified
gcp cloudbank mills Core infrastructure has been modified
gcp cloudbank palomar Core infrastructure has been modified
gcp cloudbank pasadena Core infrastructure has been modified
gcp cloudbank sjcc Core infrastructure has been modified
gcp cloudbank tuskegee Core infrastructure has been modified
gcp cloudbank avc Core infrastructure has been modified
gcp cloudbank csu Core infrastructure has been modified
aws 2i2c-aws-us researchdelight Core infrastructure has been modified
aws victor prod Core infrastructure has been modified
aws nasa-cryo prod Core infrastructure has been modified
gcp 2i2c-uk lis Core infrastructure has been modified

@yuvipanda yuvipanda changed the title Revert "Revert "Stop forcing use of NotebookApp"" Stop forcing use of NotebookApp, take 2 Feb 15, 2023
@yuvipanda yuvipanda self-assigned this Feb 15, 2023
@consideRatio
Copy link
Member

I think #2203 returned something fixed to a broken state for at least one hub, so reverting that revert resolves one issue for sure!

I didn't understand this part so well though

It still fails with the newest image:

Are you saying that merging this will still lead to issues, or that merging #2203 (reverting take 1) didn't solve the issues?

@yuvipanda
Copy link
Member Author

@consideRatio apologies, I have reworded the PR description better now. Does that make things clearer?

@consideRatio
Copy link
Member

consideRatio commented Feb 15, 2023

I reapproved, if this works go for it!

Btw it became clearer, but I was still confused about if this was in a mergable state or if it was pending some action. I guess the action was to get a new image built!

@yuvipanda
Copy link
Member Author

Ok, with 2i2c-org/2i2c-hubs-image#5 the image seems to work. I've deployed this to the staging hub on cloudbank, and it can be tested with https://staging.cloudbank.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fdorairajsanjay%2Fcis107-spring2023&urlpath=tree%2Fcis107-spring2023%2Flab%2Flab01%2Flab01.ipynb&branch=main. I've asked @sean-morris to test this out a bit before deploying this.

Here's a diff of the package versions between the current image and the new image. I think this is ok, even if long term unsustainable.

--- cur	2023-02-15 19:09:01.857447722 +0000
+++ untitled.txt	2023-02-15 19:08:47.952078859 +0000
@@ -1,257 +1,276 @@
-aiohttp==3.8.1
-aiosignal==1.2.0
-alembic==1.8.0
+aiofiles==22.1.0
+aiohttp==3.8.4
+aiosignal==1.3.1
+aiosqlite==0.18.0
+alembic==1.9.3
 altair==4.2.0
-anyio==3.6.1
+anyio==3.6.2
 argon2-cffi==21.3.0
 argon2-cffi-bindings==21.2.0
+arrow==1.2.3
 astropy==4.3.1
 astroquery==0.4.6
 async-generator==1.10
 asynctest==0.13.0
 async-timeout==4.0.2
-attrs==21.4.0
-Babel==2.10.3
+attrs==20.3.0
+Babel==2.11.0
 backcall==0.2.0
 beautifulsoup4==4.11.1
-bitarray==2.5.1
-bleach==5.0.1
-blis==0.7.8
-boto3==1.24.17
-botocore==1.27.17
-branca==0.5.0
+bitarray==2.7.2
+bleach==6.0.0
+blis==0.7.9
+boto3==1.26.71
+botocore==1.29.71
+branca==0.6.0
 brotlipy==0.7.0
 cached-property==1.5.2
-cachetools==5.2.0
-catalogue==2.0.7
-catalystcoop.pudl==0.3.2
+cachetools==5.3.0
+catalogue==2.0.8
+catalystcoop.pudl==0.2.0
 certifi==2020.6.20
 certipy==0.1.3
 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1595805535531/work
 chardet==3.0.4
-charset-normalizer==2.1.0
+charset-normalizer==3.0.1
 click==8.1.3
 click-default-group==1.2.2
 click-plugins==1.1.1
 cligj==0.7.2
-cloudpickle==2.1.0
+cloudpickle==2.2.1
 codecov==2.1.12
 coloredlogs==15.0.1
 conda==4.8.3
 conda-package-handling==1.6.0
-coverage==6.4.1
+coverage==7.1.0
 cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1595348769214/work
 cycler==0.11.0
-cymem==2.0.6
+cymem==2.0.7
 dask==2020.12.0
 dask-gateway==0.9.0
 datapackage==1.15.2
 datascience==0.17.5
 dbfread==2.0.7
-debugpy==1.6.0
+debugpy==1.6.6
 decorator==5.1.1
 defusedxml==0.7.1
-dill==0.3.5.1
+dill==0.3.6
 distributed==2020.12.0
-docker==5.0.3
-docutils==0.18.1
+docker==6.0.1
+docutils==0.19
 entrypoints==0.4
 et-xmlfile==1.1.0
-fastjsonschema==2.15.3
-Fiona==1.8.21
+exceptiongroup==1.1.0
+fastjsonschema==2.16.2
+Fiona==1.9.1
 folium==0.12.1.post1
-fonttools==4.33.3
-frozenlist==1.3.0
-future==0.18.2
+fonttools==4.38.0
+fqdn==1.5.1
+frozenlist==1.3.3
+future==0.18.3
 galpy==1.7.2
 geographiclib==1.52
 geojson==2.5.0
 geopandas==0.10.2
 geopy==2.2.0
 goodtables==2.5.4
-google-api-core==2.8.2
-google-api-python-client==2.51.0
-googleapis-common-protos==1.56.3
-google-auth==2.8.0
+google-api-core==2.11.0
+google-api-python-client==2.78.0
+googleapis-common-protos==1.58.0
+google-auth==2.16.0
 google-auth-httplib2==0.1.0
-google-auth-oauthlib==0.5.2
-greenlet==1.1.2
-gspread==5.4.0
-h3==3.7.4
+google-auth-oauthlib==1.0.0
+greenlet==2.0.2
+gspread==5.7.2
+h3==3.7.6
 HeapDict==1.0.1
 html5lib==1.1
-httplib2==0.20.4
+httplib2==0.21.0
 humanfriendly==10.0
 idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1593328102638/work
-ijson==3.1.4
-importlib-metadata==4.12.0
-importlib-resources==5.8.0
-iniconfig==1.1.1
-ipykernel==6.15.0
+ijson==3.2.0.post0
+importlib-metadata==6.0.0
+importlib-resources==5.10.2
+iniconfig==2.0.0
+ipykernel==6.16.2
 ipython==7.34.0
 ipython-genutils==0.2.0
 ipywidgets==7.5.1
 isodate==0.6.1
-jedi==0.18.1
+isoduration==20.11.0
+jaraco.classes==3.2.3
+jedi==0.18.2
 jeepney==0.8.0
 Jinja2==3.1.2
 jmespath==1.0.1
-joblib==1.1.0
-json5==0.9.8
-jsonlines==3.0.0
+joblib==1.2.0
+json5==0.9.11
+jsonlines==3.1.0
 jsonpointer==2.3
-jsonschema==4.6.0
+jsonschema==4.17.3
 jupyter-client==6.1.12
-jupyter-contrib-core==0.3.3
+jupyter-contrib-core==0.4.2
 jupyter-contrib-nbextensions==0.5.1
-jupyter-core==4.10.0
+jupyter_core==4.12.0
+jupyter-events==0.5.0
 jupyter-highlight-selected-word==0.2.0
-jupyterhub==2.3.1
-jupyterlab==3.4.3
+jupyterhub==3.1.1
+jupyterlab==3.6.1
 jupyterlab-pygments==0.2.2
-jupyterlab-server==2.14.0
+jupyterlab_server==2.19.0
 jupyterlab-widgets==1.0.2
 jupyter-latex-envs==1.4.6
-jupyter-nbextensions-configurator==0.4.1
+jupyter-nbextensions-configurator==0.6.1
 jupyter-resource-usage==0.5.0
 jupyter-rsession-proxy==1.2
-jupyter-server==1.18.0
+jupyter-server==1.23.6
+jupyter_server_fileid==0.6.0
 jupyter-server-proxy==3.0.2
+jupyter_server_ydoc==0.6.1
 jupyter-telemetry==0.1.0
 jupyter-tree-download==1.0.1
+jupyter-ydoc==0.2.2
 jupytext==1.10.3
-keyring==23.6.0
-kiwisolver==1.4.3
+keyring==23.13.1
+kiwisolver==1.4.4
 langcodes==3.3.0
 linear-tsv==1.1.0
-lxml==4.9.0
-Mako==1.2.0
+lxml==4.9.2
+Mako==1.2.4
 markdown-it-py==0.6.2
-MarkupSafe==2.1.1
+MarkupSafe==2.1.2
 matplotlib==3.5.2
-matplotlib-inline==0.1.3
-mdit-py-plugins==0.2.8
+matplotlib-inline==0.1.6
+mdit-py-plugins==0.2.6
 mistune==0.8.4
+more-itertools==9.0.0
 mpmath==1.2.1
 msgpack==1.0.4
-multidict==6.0.2
+multidict==6.0.4
 munch==2.5.0
-murmurhash==1.0.7
-nbclassic==0.3.7
+murmurhash==1.0.9
+nbclassic==0.5.1
 nbclient==0.5.13
-nbconvert==6.5.0
-nbformat==5.4.0
-nbgitpuller==0.10.2
-nest-asyncio==1.5.5
+nbconvert==6.5.4
+nbformat==5.7.3
+nbgitpuller==1.1.1
+nest-asyncio==1.5.6
 networkx==2.6.3
 nltk==3.7
 notebook==6.1.4
-notebook-shim==0.1.0
+notebook_shim==0.2.2
 numpy==1.21.6
-oauthlib==3.2.0
-openpyxl==3.0.10
+oauthlib==3.2.2
+openpyxl==3.1.1
 otter-grader==2.2.6
-packaging==21.3
+packaging==23.0
 pamela==1.0.0
 pandas==1.3.5
 pandocfilters==1.5.0
 parso==0.8.3
-pathy==0.6.1
-patsy==0.5.2
+pathy==0.10.1
+patsy==0.5.3
 pdfkit==1.0.0
 pexpect==4.8.0
 pickleshare==0.7.5
-Pillow==9.1.1
-plotly==5.9.0
+Pillow==9.4.0
+pkgutil_resolve_name==1.3.10
+plotly==5.13.0
 pluggy==1.0.0
-preshed==3.0.6
-prometheus-client==0.14.1
-prompt-toolkit==3.0.30
-protobuf==4.21.2
-psutil==5.9.1
+preshed==3.0.8
+prometheus-client==0.16.0
+prompt-toolkit==3.0.36
+protobuf==4.21.12
+psutil==5.9.4
 ptyprocess==0.7.0
-py==1.11.0
-pyarrow==8.0.0
+pyarrow==11.0.0
 pyasn1==0.4.8
 pyasn1-modules==0.2.8
-pybloom-live==3.1.0
+pybloom-live==4.0.0
 pycosat==0.6.3
 pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
 pydantic==1.8.2
 pyerfa==2.0.0.1
-Pygments==2.12.0
+Pygments==2.14.0
 pyOpenSSL==19.1.0
 pyparsing==3.0.9
-PyPDF2==2.4.0
+PyPDF2==3.0.1
 pyproj==3.2.1
-pyrsistent==0.18.1
+pyrsistent==0.19.3
 PySocks==1.7.1
-pytest==7.1.2
+pytest==7.2.1
 python-dateutil==2.8.2
-python-json-logger==2.0.2
+python-json-logger==2.0.6
 python-snappy==0.6.1
-pytz==2022.1
+pytz==2022.7.1
 pyvo==1.2.1
 PyYAML==6.0
-pyzmq==23.2.0
-regex==2022.6.2
-requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1592425495151/work
+pyzmq==25.0.0
+regex==2022.10.31
+requests==2.28.2
 requests-oauthlib==1.3.1
+rfc3339-validator==0.1.4
 rfc3986==2.0.0
+rfc3986-validator==0.1.1
 rise==5.7.1
-rsa==4.8
+rsa==4.9
 ruamel-yaml==0.15.80
 s3transfer==0.6.0
 scikit-learn==1.0.2
 scipy==1.7.3
 seaborn==0.11.2
-SecretStorage==3.3.2
+SecretStorage==3.3.3
 Send2Trash==1.8.0
-Shapely==1.8.2
+shapely==2.0.1
 simpervisor==0.4
 simpleeval==0.9.12
 six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
-smart-open==5.2.1
-sniffio==1.2.0
+smart-open==6.3.0
+sniffio==1.3.0
 sortedcontainers==2.4.0
-soupsieve==2.3.2.post1
+soupsieve==2.4
 spacy==3.3.1
-spacy-legacy==3.0.9
-spacy-loggers==1.0.2
-SQLAlchemy==1.4.39
-srsly==2.4.3
+spacy-legacy==3.0.12
+spacy-loggers==1.0.4
+SQLAlchemy==1.4.46
+srsly==2.4.5
 statistics==1.0.3.5
 statsmodels==0.13.2
 sympy==1.10.1
 tableschema==1.20.2
-tableschema-sql==1.3.2
+tableschema-sql==1.1.0
 tabulator==1.53.5
 tblib==1.7.0
-tenacity==8.0.1
-terminado==0.15.0
+tenacity==8.2.1
+terminado==0.17.1
 thinc==8.0.17
 threadpoolctl==3.1.0
 timezonefinder==6.0.1
-tinycss2==1.1.1
+tinycss2==1.2.1
 toml==0.10.2
 tomli==2.0.1
-toolz==0.11.2
-tornado==6.1
+toolz==0.12.0
+tornado==6.2
 tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1594937875116/work
-traitlets==5.3.0
-typer==0.4.1
-typing-extensions==4.1.1
+traitlets==5.9.0
+typer==0.4.2
+typing_extensions==4.1.1
 unicodecsv==0.14.1
+uri-template==1.2.0
 uritemplate==4.1.1
-urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1595434816409/work
+urllib3==1.26.14
 voila==0.2.6
-wasabi==0.9.1
-wcwidth==0.2.5
+wasabi==0.10.1
+wcwidth==0.2.6
+webcolors==1.12
 webencodings==0.5.1
-websocket-client==1.3.3
+websocket-client==1.5.1
 widgetsnbextension==3.5.2
 xlrd==2.0.1
-XlsxWriter==3.0.3
-yarl==1.7.2
+XlsxWriter==3.0.8
+xxhash==3.2.0
+yarl==1.8.2
+y-py==0.5.5
+ypy-websocket==0.8.2
 zict==2.2.0
-zipp==3.8.0
+zipp==3.13.0
\ No newline at end of file

@yuvipanda
Copy link
Member Author

Validated message from @sean-morris:

I just ran Lab 1, 4, and 8 from Spring 22. These labs include the packages: otter-grader, datascience, numpy, and matplotlib. There are definitely no import errors -- the packages seem to load fine. I ran through plotting, rendering tables, and otter-grader checks. All seems solid

Gonna merge!

@yuvipanda yuvipanda merged commit 9781aac into master Feb 16, 2023
@yuvipanda yuvipanda deleted the revert-2203-revert-2160-no-mo-nobook branch February 16, 2023 23:45
@github-actions
Copy link

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/4199215380

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants