Interactive demo of GPU rendering error

[kyleaoman]

I've attempted to try out the example notebook gpu_ray_with_widget_and_logging_v2.ipynb. The other example notebooks showcasing the GPU functionality run fine for me on the same environment so it seems like I have the GPU configured correctly. My suspicion is that this might be a jupyterlab compatibility issue. Some details:

1. I'm running on the cosma system, specifically node login8b, which is equipped with an nvidia A100 GPU (https://www.dur.ac.uk/icc/cosma/support/gpu/).
2. I'm using cuda version 11.2, which the HPC admin team has installed as a module. I'm not entirely confident about loading a module for use inside a jupyterlab environment so I queried the module (which is just a set of instructions for setting environment variables) and added a cell to the top of the notebook to set those variables through os.environ.
3. I'm working in a virtual environment with the following packages installed (everything done with pip, I used the third method to install paicos, i.e. clone repository and then install with pip):

> pip freeze
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
astropy==6.0.0
astropy-iers-data==0.2024.2.19.0.28.47
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.14.0
beautifulsoup4==4.12.3
bleach==6.1.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
comm==0.2.1
contourpy==1.2.0
cupy-cuda112==10.6.0
cycler==0.12.1
Cython==3.0.8
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
exceptiongroup==1.2.0
executing==2.0.1
fastjsonschema==2.19.1
fastrlock==0.8.2
fonttools==4.49.0
fqdn==1.5.1
h11==0.14.0
h5py==3.10.0
httpcore==1.0.3
httpx==0.26.0
idna==3.6
iniconfig==2.0.0
ipykernel==6.29.2
ipympl==0.9.3
ipython==8.21.0
ipython-genutils==0.2.0
ipywidgets==8.1.2
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.3
json5==0.9.17
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.9.0
jupyter-lsp==2.2.2
jupyter_client==8.6.0
jupyter_core==5.7.1
jupyter_server==2.12.5
jupyter_server_terminals==0.5.2
jupyterlab==4.1.2
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.3
jupyterlab_widgets==3.0.10
kiwisolver==1.4.5
llvmlite==0.42.0
MarkupSafe==2.1.5
matplotlib==3.8.3
matplotlib-inline==0.1.6
mistune==3.0.2
nbclient==0.9.0
nbconvert==7.16.1
nbformat==5.9.2
nest-asyncio==1.6.0
notebook==7.1.0
notebook_shim==0.2.4
numba==0.59.0
numpy==1.24.4
overrides==7.7.0
packaging==23.2
-e git+ssh://[email protected]/tberlok/paicos.git@4cca5893062d5435a1e6d849e3113d47e62dbd8a#egg=paicos
pandocfilters==1.5.1
parso==0.8.3
pexpect==4.9.0
pillow==10.2.0
platformdirs==4.2.0
pluggy==1.4.0
prometheus_client==0.20.0
prompt-toolkit==3.0.43
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
pyerfa==2.0.1.1
Pygments==2.17.2
pyparsing==3.1.1
pytest==8.0.1
pytest-order==1.2.0
python-dateutil==2.8.2
python-json-logger==2.0.7
PyYAML==6.0.1
pyzmq==25.1.2
qtconsole==5.5.1
QtPy==2.4.1
referencing==0.33.0
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.0
scipy==1.12.0
Send2Trash==1.8.2
six==1.16.0
sniffio==1.3.0
soupsieve==2.5
stack-data==0.6.3
terminado==0.18.0
tinycss2==1.2.1
tomli==2.0.1
tornado==6.4
traitlets==5.14.1
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
uri-template==1.3.0
urllib3==2.2.1
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
widgetsnbextension==4.0.10

4. I'm working in a jupyterlab instance hosted on the cosma jupyterhub (https://www.dur.ac.uk/icc/cosma/support/jupyterhub/). I have a suspicion that this may be the root of the problem since jupyter-notebook and jupyterlab have some compatibility issues. Might be possible to test on the TNG jupyterlab instance, but I don't know if they have any GPUs available and if so, how to use them...
5. The notebook runs fine without errors, but instead of an interactive interface at the end of the last cell I just get 11 copies of the message Error displaying widget: model not found

I can try to talk to our system admins in the coming days to see if there's a good way of starting a notebook server to use the GPUs on our system (rather than a jupyterhub/jupyterlab server) - a priori I don't know a way to do this on this system that doesn't involve an agonizingly slow remotely-hosted browser session.

[tberlok]

Hi Kyle,

I am honestly not sure how to debug these kind of things. Looking on the web, I've also noticed the frequent issues with widgets and jupyterlab. I am myself using jupyter notebooks in the browser. Here is my environment:

[berlok@astro01 ~]$ pip list 
Package                       Version      Editable project location
----------------------------- ------------ ---------------------------------
aiohttp                       3.8.5
aiosignal                     1.3.1
anyio                         4.0.0
appdirs                       1.4.4
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
astroid                       3.0.1
astropy                       5.3.3
asttokens                     2.4.0
async-lru                     2.0.4
async-timeout                 4.0.3
attrs                         23.1.0
awkward                       2.5.0
awkward-cpp                   26
Babel                         2.12.1
backcall                      0.2.0
backports.functools-lru-cache 1.6.5
beautifulsoup4                4.12.2
bleach                        6.0.0
boltons                       23.0.0
Brotli                        1.1.0
cached-property               1.5.2
cachetools                    5.3.2
certifi                       2023.7.22
cffi                          1.15.1
charset-normalizer            3.2.0
click                         8.1.7
cloudpickle                   3.0.0
cmasher                       1.6.3
cmyt                          1.4.0
colorama                      0.4.6
colorspacious                 1.1.2
comm                          0.1.4
conda                         23.7.4
conda-libmamba-solver         23.7.0
conda-package-handling        2.2.0
conda_package_streaming       0.9.0
contourpy                     1.1.1
cryptography                  41.0.4
cubinlinker-cu11              0.3.0.post1
cuda-python                   11.8.3
cudf-cu11                     23.10.2
cufinufft                     1.3
cuml-cu11                     23.10.0
cupy-cuda112                  10.6.0
cupy-knn                      0.2.5        /lustre/hpc/astro/berlok/cupy-knn
cycler                        0.11.0
Cython                        3.0.4
dask                          2023.9.2
dask-cuda                     23.10.0
dask-cudf-cu11                23.10.2
debugpy                       1.8.0
decorator                     5.1.1
defusedxml                    0.7.1
dill                          0.3.7
distributed                   2023.9.2
e13tools                      0.9.6
entrypoints                   0.4
ewah-bool-utils               1.1.0
exceptiongroup                1.1.3
executing                     1.2.0
fastjsonschema                2.18.0
fastrlock                     0.8.2
finufft                       2.1.0
flake8                        6.1.0
fonttools                     4.42.1
fortls                        2.13.0
fortran-language-server       1.12.0
fqdn                          1.5.1
frozenlist                    1.4.0
fsspec                        2023.10.0
gmpy2                         2.1.2
h5py                          3.9.0
idna                          3.4
importlib-metadata            6.8.0
importlib-resources           6.1.0
iniconfig                     2.0.0
ipykernel                     6.25.2
ipympl                        0.9.3
ipython                       8.15.0
ipython-genutils              0.2.0
ipywidgets                    8.1.1
isoduration                   20.11.0
isort                         5.12.0
jedi                          0.19.0
Jinja2                        3.1.2
joblib                        1.3.2
json5                         0.9.14
jsonpatch                     1.33
jsonpointer                   2.4
jsonschema                    4.19.1
jsonschema-specifications     2023.7.1
jupyter                       1.0.0
jupyter_client                8.3.1
jupyter-console               6.6.3
jupyter_core                  5.3.1
jupyter-events                0.7.0
jupyter-lsp                   2.2.0
jupyter_server                2.7.3
jupyter_server_terminals      0.4.4
jupyterlab                    4.0.6
jupyterlab-pygments           0.2.2
jupyterlab_server             2.25.0
jupyterlab-widgets            3.0.9
kiwisolver                    1.4.5
libmambapy                    1.5.1
llvmlite                      0.40.1
locket                        1.0.0
loguru                        0.7.2
Mako                          1.3.0
mamba                         1.5.1
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
matplotlib                    3.8.0
matplotlib-inline             0.1.6
mccabe                        0.7.0
mdurl                         0.1.2
mistune                       3.0.1
more-itertools                10.1.0
mpmath                        1.3.0
msgpack                       1.0.7
multidict                     6.0.4
munkres                       1.1.4
nbclient                      0.8.0
nbconvert                     7.8.0
nbformat                      5.9.2
nest-asyncio                  1.5.6
notebook                      7.0.4
notebook_shim                 0.2.3
numba                         0.57.1
numpy                         1.24.4
nvtx                          0.2.8
overrides                     7.4.0
packaging                     23.1
paicos                        0.1.8
pandas                        1.5.3
pandocfilters                 1.5.0
parso                         0.8.3
partd                         1.4.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        10.0.1
pip                           23.2.1
pkgutil_resolve_name          1.3.10
platformdirs                  3.10.0
pluggy                        1.3.0
ply                           3.11
pooch                         1.7.0
prometheus-client             0.17.1
prompt-toolkit                3.0.39
protobuf                      4.25.1
psutil                        5.9.5
ptxcompiler-cu11              0.7.0.post1
ptyprocess                    0.7.0
pure-eval                     0.2.2
pyarrow                       12.0.1
pycodestyle                   2.11.1
pycosat                       0.6.4
pycparser                     2.21
pycuda                        2023.1
pyerfa                        2.0.0.3
pyflakes                      3.1.0
Pygments                      2.16.1
pylibraft-cu11                23.10.0
pylint                        3.0.2
pynvml                        11.4.1
pyOpenSSL                     23.2.0
pyparsing                     3.1.1
PyQt5                         5.15.9
PyQt5-sip                     12.12.2
PySocks                       1.7.1
pytest                        7.4.3
pytest-order                  1.1.0
python-dateutil               2.8.2
python-json-logger            2.0.7
pytools                       2023.1.1
pytz                          2023.3.post1
pyvista                       0.42.2
PyYAML                        6.0.1
pyzmq                         25.1.1
qtconsole                     5.4.4
QtPy                          2.4.0
raft-dask-cu11                23.10.0
referencing                   0.30.2
requests                      2.31.0
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.7.0
rmm-cu11                      23.10.0
rpds-py                       0.10.3
ruamel.yaml                   0.17.32
ruamel.yaml.clib              0.2.7
scikit-learn                  1.3.2
scipy                         1.11.2
scooby                        0.7.3
Send2Trash                    1.8.2
setuptools                    68.2.2
sip                           6.7.11
six                           1.16.0
sniffio                       1.3.0
sortedcontainers              2.4.0
soupsieve                     2.5
stack-data                    0.6.2
sympy                         1.12
tblib                         3.0.0
terminado                     0.17.1
threadpoolctl                 3.2.0
tinycss2                      1.2.1
toml                          0.10.2
tomli                         2.0.1
tomli_w                       1.0.0
tomlkit                       0.12.1
toolz                         0.12.0
tornado                       6.3.3
tqdm                          4.66.1
traitlets                     5.10.1
treelite                      3.9.1
treelite-runtime              3.9.1
typing_extensions             4.8.0
typing-utils                  0.1.0
ucx-py-cu11                   0.34.0
unicodedata2                  15.0.0
unyt                          2.9.5
uri-template                  1.3.0
urllib3                       2.0.5
vtk                           9.2.6
wcwidth                       0.2.6
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.6.3
wheel                         0.41.2
widgetsnbextension            4.0.9
wslink                        1.11.4
xvfbwrapper                   0.2.9
yarl                          1.9.2
yt                            4.2.2
zict                          3.0.0
zipp                          3.17.0
zstandard                     0.21.0

What I have done is that I have added information about the requirements at the top of the notebooks, see e.g. here,
https://paicos.readthedocs.io/en/latest/notebooks/notebook6_interactive_visualization_on_the_gpu.html

If you still can't get it working, then I am unsure about what the next steps are as it's outside my expertise to provide user support on ipywidgets. I might consider keeping the example in the repo but removing the highlight of this capability from the JOSS paper.

The other thing I will do is to create a simple python script for the GPU timings, so that you can check the timings outside of the notebook environments.

Best,
Thomas

[kyleaoman]

I think that signposting in the notebook that a standalone notebook server is likely to work but jupyterlab might not is probably enough. I'll email my HPC support team and see if there's any way I can try using the GPUs with a standalone notebook server (without X-forwarding a browser session, tried that and it's painfully unresponsive).

[tberlok]

Thanks, I have done as suggested and added a warning about jupyter lab: https://paicos.readthedocs.io/en/latest/notebooks/notebook6_interactive_visualization_on_the_gpu.html

I have also added a high-resolution data set, so that getting it to work should be a bit more fun.