Incompatibilities between patch versions when installed in both server and kernel environment

[wholtz]

This issue was originally discussed in #34 but I now see this is a distinct issue.

If a jupyterlab user wants to make use of a custom kernel and use ipympl in their notebooks, then ipympl must be installed in both the jupyterlab environment and their kernel environment. If different versions of ipympl are installed in the jupyterlab environment and kernel environments, then sometimes ipympl breaks even for differences in the patch version. If jupyterlab is running within a multi-user environment (jupyterhub) and the user does not control the server, then the user may not know what version of ipympl is installed in the jupyterlab environment. This can make it difficult for users to determine the version of ipympl they need to install in their kernels. Additionally, the kernel environments are likely to break when upgrades occur to ipympl in the jupyterlab environment.

Test setup

The test were run using 6 builds of the following Dockerfile, where LAB_IPYMPL_VERSION modified for each build.

# this is a micromamba:0.19.1 image, using digest instead of tag for reproducibility
FROM mambaorg/micromamba@sha256:1ab2666fcd1559e9865d599433671f97cb735103af9351e5e10a3f0d5b6048c8

# version of ipympl to install into the jupyterlab environment
ARG LAB_IPYMPL_VERSION=0.8.0

# Create conda environments:
# "kernelXXX" environments will be available in jupyter notebooks
# "lab" environment will run the jupyterlab server (created at line 56)

RUN micromamba create --yes --name kernel085 --channel conda-forge \
       ipympl=0.8.5 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel084 --channel conda-forge \
       ipympl=0.8.4 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel083 --channel conda-forge \
       ipympl=0.8.3 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel082 --channel conda-forge \
       ipympl=0.8.2 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel081 --channel conda-forge \
       ipympl=0.8.1 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel080 --channel conda-forge \
       ipympl=0.8.0 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1

# activate the "kernel" environment to be active within in the next RUN command
ARG MAMBA_DOCKERFILE_ACTIVATE=1 

# create a kernel.json file for each environment
ENV ENV_NAME=kernel085
RUN python -m ipykernel install --user --name kernel085 --display-name kernel085
ENV ENV_NAME=kernel084
RUN python -m ipykernel install --user --name kernel084 --display-name kernel084
ENV ENV_NAME=kernel083
RUN python -m ipykernel install --user --name kernel083 --display-name kernel083
ENV ENV_NAME=kernel082
RUN python -m ipykernel install --user --name kernel082 --display-name kernel082
ENV ENV_NAME=kernel081
RUN python -m ipykernel install --user --name kernel081 --display-name kernel081
ENV ENV_NAME=kernel080
RUN python -m ipykernel install --user --name kernel080 --display-name kernel080

RUN micromamba create --yes --name lab --channel conda-forge \
       ipympl=$LAB_IPYMPL_VERSION \
       jupyterlab=3.2.6 \
       jupyterlab_widgets=1.0.2 \
       python=3.10.1 \
       xorg-libxrender=0.9.10 && \
    micromamba clean --all --yes

# activate the "lab" environment for when the CMD executes
ENV ENV_NAME=lab

# RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager [email protected]

EXPOSE 8888
WORKDIR /home/micromamba
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--ServerApp.token=''", "--ServerApp.root_dir=/home/micromamba"]

For each build-kernel pair, I ran the following in a notebook:

import matplotlib.pyplot as plt
%matplotlib widget
fig = plt.figure()
plt.plot([1,2,7])

After selecting the kernel for the next test, I performed a 'Empty Cache and Hard Reload' with the the Chrome web browser.

Versions

Here is the version info for the test when ipympl v0.8.0 was installed in the jupyterlab environment:

python -c "import sys; print('\n',sys.version); import ipympl; print('ipympl version:', ipympl.__version__)" && jupyter --version && jupyter nbextension list && jupyter labextension list

 3.10.1 | packaged by conda-forge | (main, Dec 22 2021, 01:39:05) [GCC 9.4.0]
ipympl version: 0.8.0
Selected Jupyter core packages...
IPython          : 8.0.0
ipykernel        : 6.7.0
ipywidgets       : 7.6.5
jupyter_client   : 7.1.0
jupyter_core     : 4.9.1
jupyter_server   : 1.13.2
jupyterlab       : 3.2.6
nbclient         : 0.5.9
nbconvert        : 6.4.0
nbformat         : 5.1.3
notebook         : 6.4.7
qtconsole        : not installed
traitlets        : 5.1.1
Known nbextensions:
  config dir: /opt/conda/envs/lab/etc/jupyter/nbconfig
    notebook section
      jupyter-matplotlib/extension  enabled 
      - Validating: OK
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
JupyterLab v3.2.6
/opt/conda/envs/lab/share/jupyter/labextensions
        jupyter-matplotlib v0.10.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)

For the other tests, all these values except for the ipympl version and the jupyter-matplotlib remained constant. jupyter-matplotlib was never directly installed, it always came from the installation of ipympl.

Results

I ran a series of tests to determine which combinations of the ipympl v0.8.x packages are functional when installed in both the jupyterlab environment and the kernel environment. Three different outputs occurred:

1. "model not found" - the text "Error displaying widget: model not found" was displayed
2. "2 button widget": the following was displayed:
   
3. "functional" - a full interactive plot with the expected toolbars was displayed

The results are summarized in this table:

Conclusions

ipympl is fragile when installed in kernels in a different environment from the jupyterlab environment. It would be great if in these types of setups, ipympl was compatible with itself across patch versions (ie, within the v0.8.x series).

Thanks!

Sorry this was so long. This module is an important and appreciated part of my workflows. Thanks you!

[martinRenou]

Thanks for the detailed report :)

Indeed, when the versions don't match it will fail to render (even if, in some cases, it could technically work). This check is being done by the ipywidgets's JupyterLab Widget Manager.

Currently we have a:

• back-end version (ipympl, currently 0.8.5)
• front-end version (jupyter-matplotlib currently 0.10.4)

Technically, we should also have a "model version", a version of some kind of protocol that the back-end/front-end speaks. I think this would be the cleanest way of doing it (and it is possible to do that with ipywidgets), but I don't know any widget which does that.
This would allow e.g. ipympl 0.8.5 and jupyter-matplotlib 0.9.1 to be compatible if their model versions matches.

I am personally totally open to start doing this. Although I am not sure what the transition would look like, and it would not affect old ipympl versions.

[martinRenou]

We would bump such "model version" only if there is a breaking change in the widget model (e.g. if we rename a widget property in the model). It's a bit more maintenance pain but it might be worth it.

It might actually reduce the number of issues "it fails to render in JupyterLab with model not found" that we see popping on all widgets libs.

[ianhi]

It might actually reduce the number of issues "it fails to render in JupyterLab with model not found" that we see popping on all widgets libs.

Agreed I think it would.

We would bump such "model version" only if there is a breaking change in the widget model (e.g. if we rename a widget property in the model). It's a bit more maintenance pain but it might be worth it.

but I don't know any widget which does that.

I agree that this is good. But I think the only example is actually ipywidgets itself:
https://github.com/jupyter-widgets/ipywidgets/blob/f7c48afb27bd80b14966b862ad03e30c40c7481f/python/ipywidgets/ipywidgets/_version.py#L9-L12

[ianhi]

We actually already have some of the infrastructure for this. I think that so long as:

ipympl/ipympl/_version.py

Line 3 in 6a7c2ba

js_semver = '^0.10.5'

matches these:

ipympl/src/mpl_widget.ts

Lines 33 to 34 in 6a7c2ba

	_model_module_version: '^' + MODULE_VERSION,
	_view_module_version: '^' + MODULE_VERSION,

then things should work.

So we need to decouple that number from the package versioning. This is what ipywidgets does: https://github.com/jupyter-widgets/ipywidgets/blob/88cec8be2869e75760a6a99bba129e6011d9043f/python/ipywidgets/ipywidgets/_version.py

So I propose that we take the current protocol version and call it 1.0.0 as this isn't really a public facing thing and I think this will make the future version comparisons easier to understand.

[ianhi]

oh actually, changing it is worse. We should leave it exactly the same until we actually change the protocol. So we should decouple js_semver from the package.json version, and also change the _model_module_version to not follow the package.json number

[ianhi]

I'll try this as a follow up to #446 and then on release we can see if the situation improved. Then we can leave extensive comments to that effect and/or add this as docs to contributing guide (and to ipywidgets) so we know what to do in the future.

[martinRenou]

We should leave it exactly the same until we actually change the protocol. So we should decouple js_semver from the package.json version, and also change the _model_module_version to not follow the package.json number

Yes! Also making this version 1.0.0 makes sense to me 👍🏽

Happy to give it a review!

[ianhi]

I think this will have been fixed going forward by #448 but let's definitely reopen if we discover that it was not

[ianhi]

heads up @wholtz that there were some issues with pbulishing the npm version so the new version 0.9.0 does not have a corresponding version on npm yet see #458

[yashkurkure]

I have this same issue at the moment. I don't know what version of ipympl is installed on the server side. Is anyone else still facing the issue?