Incompatible with Jupytext plugin (notebook corruption)

[goerz]

Description

The jupyter-collaboration plugin appears to be incompatible with the jupytext plugin. It is unclear which plugin's "fault" this is.

See mwouts/jupytext#1124

Reproduce

As described in mwouts/jupytext#1124 (comment):

1. Create a test notebook with the content print('test')
2. Pair this with jupytext to a corresponding script file (e.g., light, OP used percent)
3. Close and shutdown the kernel.
4. Open the .py file in Jupyter Lab and execute the first cell.
5. Close and shutdown the kernel.
6. Open the original .ipynb in Jupyter Lab.

Context

• Operating System and version: Linux Ubuntu 20.04
• Browser and version: Safari/Chrome/Arq (not browser dependent)
• Jupyter was installed via Miniforge 23.1-1: conda install jupyterlab notebook jupyterlab_execute_time nbdime jupytext ipywidgets ipyparallel jupyter-collaboration
• JupyterLab version: 4.0.8
• Jupytext version: 1.15.2
• Jupyter-collaboration version: 1.2.0

Troubleshoot Output

$PATH:
        /home/goerz/.pyenv/versions/miniforge3-23.3.1-1/bin
        /home/goerz/.pyenv/libexec
        /home/goerz/.pyenv/plugins/python-build/bin
        /home/goerz/.pyenv/plugins/pyenv-virtualenv/bin
        /home/goerz/bin
        /home/goerz/.pyenv/shims
        /home/goerz/.julia/juliaup//bin
        /home/goerz/.pyenv/bin
        /home/goerz/.local/bin
        /home/goerz/bin
        /home/goerz/.julia/juliaup//bin
        /home/goerz/.pyenv/bin
        /home/goerz/.local/bin
        /usr/local/sbin
        /usr/local/bin
        /usr/sbin
        /usr/bin
        /sbin
        /bin
        /usr/games
        /usr/local/games
        /snap/bin
sys.path:

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/bin

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/lib/python310.zip

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/lib/python3.10

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/lib/python3.10/lib-dynload

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/lib/python3.10/site-packages
sys.executable:

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/bin/python
sys.version:

3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0]
platform.platform():

Linux-5.13.0-37-generic-x86_64-with-glibc2.31
which -a jupyter:

/home/goerz/.pyenv/versions/miniforge3-23.3.1-1/bin/jupyter

/home/goerz/.pyenv/shims/jupyter

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[jtpio]

Thanks @goerz for the report.

Would you be able to try with the new 2.0.0 release, to see if it might fix the issue? https://github.com/jupyterlab/jupyter-collaboration/releases/tag/v2.0.0

[goerz]

I don't seem to be able to install the new release:

Edit: never mind, got it working on Linux (will try to test for the bug next)

:> conda install jupyter-collaboration==2.0.0
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides pycrdt >=0.7.0,<0.8.0 needed by pycrdt-websocket-0.12.5-pyhd8ed1ab_0
  - cannot install both pin-1-1 and pin-1-1

Could not solve for environment specs
The following packages are incompatible
├─ jupyter-collaboration 2.0.0  is not installable because it requires
│  └─ pycrdt-websocket >=0.12.5,<0.13.0 , which requires
│     └─ pycrdt >=0.7.0,<0.8.0 , which does not exist (perhaps a missing channel);
└─ pin-1 is installable with the potential options
   ├─ pin-1 1, which can be installed;
   └─ pin-1 1 conflicts with any installable versions previously reported.

Pins seem to be involved in the conflict. Currently pinned specs:
 - python 3.10.* (labeled as 'pin-1')

[jtpio]

I don't seem to be able to install the new release:

cc @davidbrochart as this seems related to the recent pycrdt change

[goerz]

Unfortunately, the issue persists in version 2.0, with the same steps to reproduce.

I've tested this on Linux in a fresh conda environment:

environment.yml

name: jupytest
channels:
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - aiosqlite=0.19.0=pyhd8ed1ab_0
  - alsa-lib=1.2.10=hd590300_0
  - anyio=4.1.0=pyhd8ed1ab_0
  - archspec=0.2.2=pyhd8ed1ab_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py310h2372a71_4
  - arrow=1.3.0=pyhd8ed1ab_0
  - asttokens=2.4.1=pyhd8ed1ab_0
  - async-lru=2.0.4=pyhd8ed1ab_0
  - attr=2.5.1=h166bdaf_1
  - attrs=23.1.0=pyh71513ae_1
  - babel=2.14.0=pyhd8ed1ab_0
  - beautifulsoup4=4.12.2=pyha770c72_0
  - black=23.11.0=py310hff52083_0
  - bleach=6.1.0=pyhd8ed1ab_0
  - boltons=23.0.0=pyhd8ed1ab_0
  - brotli=1.1.0=hd590300_1
  - brotli-bin=1.1.0=hd590300_1
  - brotli-python=1.1.0=py310hc6cd4ac_1
  - bzip2=1.0.8=hd590300_5
  - c-ares=1.23.0=hd590300_0
  - ca-certificates=2023.11.17=hbcca054_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cairo=1.18.0=h3faef2a_0
  - certifi=2023.11.17=pyhd8ed1ab_0
  - cffi=1.16.0=py310h2fee648_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - comm=0.1.4=pyhd8ed1ab_0
  - conda=23.10.0=py310hff52083_1
  - conda-libmamba-solver=23.12.0=pyhd8ed1ab_0
  - conda-package-handling=2.2.0=pyh38be061_0
  - conda-package-streaming=0.9.0=pyhd8ed1ab_0
  - contourpy=1.2.0=py310hd41b1e2_0
  - cycler=0.12.1=pyhd8ed1ab_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.8.0=py310hc6cd4ac_1
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - exceptiongroup=1.2.0=pyhd8ed1ab_0
  - executing=2.0.1=pyhd8ed1ab_0
  - expat=2.5.0=hcb278e6_1
  - fmt=10.1.1=h00ab1b0_1
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_1
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.46.0=py310h2372a71_0
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - gettext=0.21.1=h27087fc_0
  - gitdb=4.0.11=pyhd8ed1ab_0
  - gitpython=3.1.40=pyhd8ed1ab_0
  - glib=2.78.3=hfc55251_0
  - glib-tools=2.78.3=hfc55251_0
  - gmp=6.3.0=h59595ed_0
  - gmpy2=2.1.2=py310h3ec546c_1
  - graphite2=1.3.13=h58526e2_1001
  - gst-plugins-base=1.22.7=h8e1006c_1
  - gstreamer=1.22.7=h98fc4e7_1
  - harfbuzz=8.3.0=h3d44ed6_0
  - icu=73.2=h59595ed_0
  - idna=3.6=pyhd8ed1ab_0
  - importlib-metadata=7.0.0=pyha770c72_0
  - importlib_metadata=7.0.0=hd8ed1ab_0
  - importlib_resources=6.1.1=pyhd8ed1ab_0
  - ipykernel=6.26.0=pyhf8b6a83_0
  - ipyparallel=8.6.1=pyhd8ed1ab_0
  - ipython=8.18.1=pyh707e725_3
  - ipywidgets=8.1.1=pyhd8ed1ab_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - isort=5.13.2=pyhd8ed1ab_0
  - jedi=0.19.1=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - json5=0.9.14=pyhd8ed1ab_0
  - jsonpatch=1.33=pyhd8ed1ab_0
  - jsonpointer=2.4=py310hff52083_3
  - jsonschema=4.20.0=pyhd8ed1ab_0
  - jsonschema-specifications=2023.11.2=pyhd8ed1ab_0
  - jsonschema-with-format-nongpl=4.20.0=pyhd8ed1ab_0
  - jupyter-collaboration=2.0.0=pyhd8ed1ab_0
  - jupyter-lsp=2.2.1=pyhd8ed1ab_0
  - jupyter-server-mathjax=0.2.6=pyh5bfe37b_1
  - jupyter_client=8.6.0=pyhd8ed1ab_0
  - jupyter_core=5.5.0=py310hff52083_0
  - jupyter_events=0.9.0=pyhd8ed1ab_0
  - jupyter_server=2.12.1=pyhd8ed1ab_0
  - jupyter_server_fileid=0.9.0=pyhd8ed1ab_0
  - jupyter_server_terminals=0.5.0=pyhd8ed1ab_0
  - jupyter_ydoc=2.0.0=pyhd8ed1ab_0
  - jupyterlab=4.0.9=pyhd8ed1ab_0
  - jupyterlab_execute_time=3.1.0=pyhd8ed1ab_0
  - jupyterlab_pygments=0.3.0=pyhd8ed1ab_0
  - jupyterlab_server=2.25.2=pyhd8ed1ab_0
  - jupyterlab_widgets=3.0.9=pyhd8ed1ab_0
  - jupytext=1.16.0=pyhd8ed1ab_0
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.5=py310hd41b1e2_1
  - krb5=1.21.2=h659d440_0
  - lame=3.100=h166bdaf_1003
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libarchive=3.7.2=h2aa1ff5_1
  - libblas=3.9.0=20_linux64_openblas
  - libbrotlicommon=1.1.0=hd590300_1
  - libbrotlidec=1.1.0=hd590300_1
  - libbrotlienc=1.1.0=hd590300_1
  - libcap=2.69=h0f662aa_0
  - libcblas=3.9.0=20_linux64_openblas
  - libclang=15.0.7=default_hb11cfb5_4
  - libclang13=15.0.7=default_ha2b6cf4_4
  - libcups=2.3.3=h4637d8d_4
  - libcurl=8.5.0=hca28451_0
  - libdeflate=1.19=hd590300_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libexpat=2.5.0=hcb278e6_1
  - libffi=3.4.2=h7f98852_5
  - libflac=1.4.3=h59595ed_0
  - libgcc-ng=13.2.0=h807b86a_3
  - libgcrypt=1.10.3=hd590300_0
  - libgfortran-ng=13.2.0=h69a702a_3
  - libgfortran5=13.2.0=ha4646dd_3
  - libglib=2.78.3=h783c2da_0
  - libgomp=13.2.0=h807b86a_3
  - libgpg-error=1.47=h71f35ed_0
  - libiconv=1.17=hd590300_1
  - libjpeg-turbo=3.0.0=hd590300_1
  - liblapack=3.9.0=20_linux64_openblas
  - libllvm15=15.0.7=hb3ce162_4
  - libmamba=1.5.3=had39da4_2
  - libmambapy=1.5.3=py310h39ff949_2
  - libnghttp2=1.58.0=h47da74e_1
  - libnsl=2.0.1=hd590300_0
  - libogg=1.3.4=h7f98852_1
  - libopenblas=0.3.25=pthreads_h413a1c8_0
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.39=h753d276_0
  - libpq=16.1=h33b98f1_7
  - libsndfile=1.2.2=hc60ed4a_1
  - libsodium=1.0.18=h36c2ea0_1
  - libsolv=0.7.27=hfc55251_0
  - libsqlite=3.44.2=h2797004_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx-ng=13.2.0=h7e041cc_3
  - libsystemd0=255=h3516f8a_0
  - libtiff=4.6.0=ha9c0a0a_2
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.46.0=hd590300_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp-base=1.3.2=hd590300_0
  - libxcb=1.15=h0b41bf4_0
  - libxkbcommon=1.6.0=hd429924_1
  - libxml2=2.12.3=h232c23b_0
  - libzlib=1.2.13=hd590300_5
  - lz4-c=1.9.4=hcb278e6_0
  - lzo=2.10=h516909a_1000
  - mamba=1.5.3=py310h51d5547_2
  - markdown=3.5.1=pyhd8ed1ab_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markdown-kernel=0.2.1=py_0
  - markupsafe=2.1.3=py310h2372a71_1
  - matplotlib=3.8.2=py310hff52083_0
  - matplotlib-base=3.8.2=py310h62c0568_0
  - matplotlib-inline=0.1.6=pyhd8ed1ab_0
  - mdit-py-plugins=0.4.0=pyhd8ed1ab_0
  - mdurl=0.1.0=pyhd8ed1ab_0
  - mistune=3.0.2=pyhd8ed1ab_0
  - mpc=1.3.1=hfe3b2da_0
  - mpfr=4.2.1=h9458935_0
  - mpg123=1.32.3=h59595ed_0
  - mpmath=1.3.0=pyhd8ed1ab_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mypy_extensions=1.0.0=pyha770c72_0
  - mysql-common=8.0.33=hf1915f5_6
  - mysql-libs=8.0.33=hca2cd23_6
  - nbclient=0.8.0=pyhd8ed1ab_0
  - nbconvert-core=7.12.0=pyhd8ed1ab_0
  - nbdime=4.0.1=pyhd8ed1ab_0
  - nbformat=5.9.2=pyhd8ed1ab_0
  - ncurses=6.4=h59595ed_2
  - nest-asyncio=1.5.8=pyhd8ed1ab_0
  - nodejs=20.9.0=hb753e55_0
  - notebook=7.0.6=pyhd8ed1ab_0
  - notebook-shim=0.2.3=pyhd8ed1ab_0
  - nspr=4.35=h27087fc_0
  - nss=3.95=h1d7d5a4_0
  - numpy=1.26.2=py310hb13e2d6_0
  - openjpeg=2.5.0=h488ebb8_3
  - openssl=3.2.0=hd590300_1
  - overrides=7.4.0=pyhd8ed1ab_0
  - packaging=23.2=pyhd8ed1ab_0
  - pandoc=3.1.3=h32600fe_0
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - parso=0.8.3=pyhd8ed1ab_0
  - pathspec=0.12.1=pyhd8ed1ab_0
  - pcre2=10.42=hcad00b1_0
  - pexpect=4.8.0=pyh1a96a4e_2
  - pickleshare=0.7.5=py_1003
  - pillow=10.1.0=py310h01dd4db_0
  - pip=23.3.1=pyhd8ed1ab_0
  - pixman=0.42.2=h59595ed_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - platformdirs=4.1.0=pyhd8ed1ab_0
  - pluggy=1.3.0=pyhd8ed1ab_0
  - ply=3.11=py_1
  - prometheus_client=0.19.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.42=pyha770c72_0
  - psutil=5.9.5=py310h2372a71_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pulseaudio-client=16.1=hb77b528_5
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - pybind11-abi=4=hd8ed1ab_3
  - pycosat=0.6.6=py310h2372a71_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pycrdt=0.7.2=py310hcb5633a_0
  - pycrdt-websocket=0.12.5=pyhd8ed1ab_0
  - pygments=2.17.2=pyhd8ed1ab_0
  - pyparsing=3.1.1=pyhd8ed1ab_0
  - pyqt=5.15.9=py310h04931ad_5
  - pyqt5-sip=12.12.2=py310hc6cd4ac_5
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.10.13=hd12c33a_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-fastjsonschema=2.19.0=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python_abi=3.10=4_cp310
  - pytz=2023.3.post1=pyhd8ed1ab_0
  - pyyaml=6.0.1=py310h2372a71_1
  - pyzmq=25.1.2=py310h795f18f_0
  - qt-main=5.15.8=h450f30e_18
  - qutip=4.7.3=py310hcb52e73_1
  - readline=8.2=h8228510_1
  - referencing=0.32.0=pyhd8ed1ab_0
  - reproc=14.2.4.post0=hd590300_1
  - reproc-cpp=14.2.4.post0=h59595ed_1
  - requests=2.31.0=pyhd8ed1ab_0
  - rfc3339-validator=0.1.4=pyhd8ed1ab_0
  - rfc3986-validator=0.1.1=pyh9f0ad1d_0
  - rpds-py=0.13.2=py310hcb5633a_0
  - ruamel.yaml=0.18.5=py310h2372a71_0
  - ruamel.yaml.clib=0.2.7=py310h2372a71_2
  - scipy=1.11.4=py310hb13e2d6_0
  - send2trash=1.8.2=pyh41d4057_0
  - setuptools=68.2.2=pyhd8ed1ab_0
  - sip=6.7.12=py310hc6cd4ac_0
  - six=1.16.0=pyh6c4a22f_0
  - smmap=5.0.0=pyhd8ed1ab_0
  - sniffio=1.3.0=pyhd8ed1ab_0
  - soupsieve=2.5=pyhd8ed1ab_1
  - stack_data=0.6.2=pyhd8ed1ab_0
  - sympy=1.12=pypyh9d50eac_103
  - terminado=0.18.0=pyh0d859eb_0
  - tinycss2=1.2.1=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - tornado=6.3.3=py310h2372a71_1
  - tqdm=4.66.1=pyhd8ed1ab_0
  - traitlets=5.14.0=pyhd8ed1ab_0
  - truststore=0.8.0=pyhd8ed1ab_0
  - types-python-dateutil=2.8.19.14=pyhd8ed1ab_0
  - typing-extensions=4.9.0=hd8ed1ab_0
  - typing_extensions=4.9.0=pyha770c72_0
  - typing_utils=0.1.0=pyhd8ed1ab_0
  - tzdata=2023c=h71feb2d_0
  - unicodedata2=15.1.0=py310h2372a71_0
  - uri-template=1.3.0=pyhd8ed1ab_0
  - urllib3=2.1.0=pyhd8ed1ab_0
  - wcwidth=0.2.12=pyhd8ed1ab_0
  - webcolors=1.13=pyhd8ed1ab_0
  - webencodings=0.5.1=pyhd8ed1ab_2
  - websocket-client=1.7.0=pyhd8ed1ab_0
  - wheel=0.42.0=pyhd8ed1ab_0
  - widgetsnbextension=4.0.9=pyhd8ed1ab_0
  - xcb-util=0.4.0=hd590300_1
  - xcb-util-image=0.4.0=h8ee46fc_1
  - xcb-util-keysyms=0.4.0=h8ee46fc_1
  - xcb-util-renderutil=0.3.9=hd590300_1
  - xcb-util-wm=0.4.1=h8ee46fc_1
  - xkeyboard-config=2.40=hd590300_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.1.1=hd590300_0
  - xorg-libsm=1.2.4=h7391055_0
  - xorg-libx11=1.8.7=h8ee46fc_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxrender=0.9.11=hd590300_0
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xf86vidmodeproto=2.3.1=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yaml-cpp=0.8.0=h59595ed_0
  - zeromq=4.3.5=h59595ed_0
  - zipp=3.17.0=pyhd8ed1ab_0
  - zlib=1.2.13=hd590300_5
  - zstandard=0.22.0=py310h1275a96_0
  - zstd=1.5.5=hfc55251_0
prefix: /home/goerz/.pyenv/versions/miniforge3-23.3.1-1/envs/jupytest

[EduardDurech]

I'm having this issue with jupyter-collaboration==2.0.3 #162 (comment)

[mwouts]

I have had a quick look at this, and I confirm that I can reproduce the issue.

1. I created a notebook jupyter_collaboration.ipynb

[I 2024-03-24 18:54:33.716 ServerApp] Saving the content from room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:54:33.719 YDocExtension] Saving file: jupyter_collaboration.ipynb
[I 2024-03-24 18:54:33.724 ServerApp] Saving jupyter_collaboration.ipynb

2. Then I paired it to the py:percent format (File->Jupytext->Pair with percent format, then save)

[I 2024-03-24 18:55:12.392 ServerApp] Saving the content from room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:55:12.395 YDocExtension] Saving file: jupyter_collaboration.ipynb
[I 2024-03-24 18:55:12.413 ServerApp] Saving jupyter_collaboration.ipynb
[I 2024-03-24 18:55:12.447 ServerApp] Saving jupyter_collaboration.py in format py:percent

3. I closed and then re-opened the .ipynb document:

[I 2024-03-24 18:55:53.324 ServerApp] Starting buffering for 98ea9cbe-7609-4865-9f6e-733de9af1621:6e18dfb5-3b2a-494b-9006-f64f6d6e8602
[I 2024-03-24 18:55:53.327 ServerApp] Cleaning room: json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:55:54.842 ServerApp] Request for Y document 'jupyter_collaboration.ipynb' with room ID: 8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:56:00.408 YDocExtension] Processed 16 Y patches in one minute
[I 2024-03-24 18:56:00.409 YDocExtension] Connected Y users: 1
[I 2024-03-24 18:56:53.329 ServerApp] Deleting Y document from memory: json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:56:53.329 ServerApp] Room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd deleted
[I 2024-03-24 18:56:53.330 ServerApp] Deleting file jupyter_collaboration.ipynb
[I 2024-03-24 18:56:53.333 YDocExtension] Creating FileLoader for: jupyter_collaboration.ipynb
[I 2024-03-24 18:56:53.334 YDocExtension] Watching file: jupyter_collaboration.ipynb
[I 2024-03-24 18:56:53.336 ServerApp] Initializing room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd
[I 2024-03-24 18:56:53.374 ServerApp] Reading SOURCE from jupyter_collaboration.py
[I 2024-03-24 18:56:53.417 ServerApp] Content in room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd loaded from the ystore SQLiteYStore
[I 2024-03-24 18:56:53.418 ServerApp] Content in file jupyter_collaboration.ipynb is out-of-sync with the ystore SQLiteYStore
[I 2024-03-24 18:56:53.418 ServerApp] Content in room json:notebook:8eba183d-2923-4c6a-838e-e8cdcaf223cd loaded from file jupyter_collaboration.ipynb
[I 2024-03-24 18:56:53.493 ServerApp] Connecting to kernel 98ea9cbe-7609-4865-9f6e-733de9af1621.
[I 2024-03-24 18:57:00.410 YDocExtension] Processed 8 Y patches in one minute

Then my notebook has a single code cell with this content:

IyAtLS0KIyBqdXB5dGVyOgojICAganVweXRleHQ6CiMgICAgIGZvcm1hdHM6IGlweW5iLHB5OnBl
cmNlbnQKIyAgICAgdGV4dF9yZXByZXNlbnRhdGlvbjoKIyAgICAgICBleHRlbnNpb246IC5weQoj
ICAgICAgIGZvcm1hdF9uYW1lOiBwZXJjZW50CiMgICAgICAgZm9ybWF0X3ZlcnNpb246ICcxLjMn
CiMgICAgICAganVweXRleHRfdmVyc2lvbjogMS4xNi4xCiMgICBrZXJuZWxzcGVjOgojICAgICBk
aXNwbGF5X25hbWU6IFB5dGhvbiAzIChpcHlrZXJuZWwpCiMgICAgIGxhbmd1YWdlOiBweXRob24K
IyAgICAgbmFtZTogcHl0aG9uMwojIC0tLQoKIyAlJQoxICsgMQoKIyAlJSBbbWFya2Rvd25dCiMg
bWFya2Rvd24KCiMgJSUK

I confirm that text in the first cell of the notebook is a b64 encoding of the text notebook:

text = """IyAtLS0K(...)"""
print(base64.b64decode("".join(text.splitlines())).decode('utf8'))

# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.16.1
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# %%
1 + 1

# %% [markdown]
# markdown

# %%

Interestingly, this base64 encoded content is what is returned by the parent contents manager (here, a LargeFileContentsManager as Jupytext does not support async CM yet).

I think it's a bit odd that the parent CM would return a base64 encoded value, so that's maybe what I would suggest to fix. @fcollonval do you think this could be related to using the non-async CM? (I mean, possibly Jupyter-collaboration has been tested focusing on the async contents manager?)

[Sieboldianus]

This issue is pretty bad, as

1. it does not allow switching between jupytext and RTC. Notebooks will be rendered unusable
2. It is not possible to simply disable jupytext or collaboration-extension, one needs to uninstall them to allow compatibility

I tried disabling:

jupyter labextension disable @jupyter/collaboration-extension
# or
jupyter labextension disable jupyterlab-jupytext

But this does not work either way. The RTC is shown as disabled, but I still see RTC in the path for notebooks.

[davidbrochart]

RTC can be disabled with jupyter lab --YDocExtension.disable_rtc=True.

[Sieboldianus]

Thank you @davidbrochart, but after testing, it seems --YDocExtension.disable_rtc=True does not work. After starting jupyter lab with this flag, I still see:

and:

Disabling the extension does not work, either:

jupyter labextension disable @jupyter/collaboration-extension

You can see that RTC is still written in the address bar, but the extension is listed as disabled:

Based on the commit, it seems that it must be written --YDocExtension.disableRTC=true, which disables the extension but still shows RTC in the address bar for open notebooks.

Some background: The reason this is important to me is, we are building a Jupyter Lab docker container where users can choose to enable or disable features on runtime. This means that the extension should be available by default, but only enabled for those who wish to use collaboration or jupytext (e.g.).

[davidbrochart]

jupyter lab --YDocExtension.disable_rtc=True disables real-time collaboration, but the drive is still a "collaborative drive", meaning that it has the RTC: prefix. These two PRs will allow to get rid of the RTC: prefix:

• Allow customizing the ServiceManager with plugins jupyterlab#16794
• Add content providers jupyterlab#16744

[Sieboldianus]

Great, thank you. The jupytext incompatibility requires removing the RTC: prefix, as far as my tests have shown.

[goerz]

I'm unclear on whether the two PRs linked in #214 (comment) are moving towards addressing the issue. Could someone elaborate on what these do?

Based on jupyterlab/jupyterlab#14516, it would seem like the jupypter_collaboration plugin is the only way to be able to edit a Jupytext-linked .py file and have JupyterLab automatically reload the associated .ipynb. Without that, it is extremely easy to accidentally change both the .py file in an external editor and the .ipynb in JupyterLab, resulting in a conflict. For an optimal workflow, these two plugins really should work together. So I hope there can be some movement on getting this to work, eventually.

[choldgraf]

There's been some discussion about RTC and jupytext in this zulip discussion. Just cross-referencing for discoverability:

https://jupyter.zulipchat.com/#narrow/stream/469762-jupyterlab/topic/Persistent.20data.20loss.20issues.20with.20RTC.2E.2E.2E

[fperez]

Thx @choldgraf - I also flagged it on the MyST discord, as @cboettig had flagged some cool MyST/jupytext examples and I'm worried people will easily install jupytext on a whim to test it and will end up losing data. I had it happen to me last week :(

[fperez]

I think that given:

• the result can be catastrophic data loss
• the mere presence of juptyext in the environment is sufficient to cause it

That the RTC extension should try to detect the existence of jupytext's CM override and immediately either abort or provide a very visible warning to the user that data loss is possible (likely? certain?).

We should not be silently destroying user data under any circumstances. This is not about blaming either jupytext nor jupyter-rtc, it's simply about watching out for our users first. If we're destroying their data, that's a problem, and we need to take aggressive measures to avoid that, period.

Destroying a user's work is the best way to tell them to never use Jupyter again.

[mwouts]

Hi all, yes I do find this issue very concerning as well. I do care a lot for people's notebooks and Jupytext usually provides strong guaranties re notebook round trips between text and ipynb formats.

I looked a bit more into the issue (see also my comment above from March). This time I think that I have identified the problem. When jupyter-collaboration is installed the value for the format argument passed to the CM is "json". If I change it to "text" or to None then I don't get that problem anymore.

I don't find much information on the possible values for that format arguments (is None more correct than "text"?). Can I suggest that the contents managers raise an error when they are called with an incorrect value (e.g. "json" here for a text file) rather than returning an encoded content?

On Jupytext's end I have prepared a tentative fix (v1.16.5) that can be installed with

HATCH_BUILD_HOOKS_ENABLE=true pip install git+https://github.com/mwouts/jupytext.git@fix_notebook_corruption_with_jupyter_collaboration

I would love if someone could test it with Jupyter-collaboration and confirm that it works fine for them. On my end I will fix the CI and make sure that the change is not breaking for people who don't have jupyter-collaboration.

[davidbrochart]

When jupyter-collaboration is installed the value for the format argument passed to the CM is "json"

Could you link to the relevant code?

[mwouts]

Looking into this a bit more I think this bug is a conjunction of three factors:

1. Jupyter collaboration seems to pass a format="json" when calling the get method of the contents manager. @davidbrochart I am not sure exactly where this is done, but I have seen that value when debugging. Also there are mentions in this project that the model object for notebooks should always have format="json" (see screenshot below). I don't think that Jupyter Lab or Notebook pass format="json" like this (well else we would have seen the same problem...)
2. Jupytext<=1.16.5 passes down the format="json" argument from the get call for the .ipynb file on the Jupytext contents manager to the get call for the .py file on the LargeFileContentsManager.
3. The LargeFileContentsManager returns the base64 encoded version of the file if format is neither of None, "text", or "byte" (or if it can't convert the content to unicode)

Fixing any of the three items would fix the bug. I plan to address 2. soon with the linked PR. I think that 3. sounds a little fragile and I would be in favor of a stricter approach (e.g. error if unicode conversion fails or if format is not one of the expected values) but I understand that a stricter implementation could have unintended impacts. Finally re 1. I do agree that .ipynb notebooks are in the "json" format but possibly Jupyter collaboration does not need to differ from Jupyter Lab in how it calls the contents manager?

[davidbrochart]

I just tried using Jupytext with RTC, and I get this error:

    self._contents_manager.get(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.path,
        ^^^^^^^^^^
        content=False,
        ^^^^^^^^^^^^^^
        require_hash=True,
        ^^^^^^^^^^^^^^^^^^
    )
    ^
TypeError: build_jupytext_contents_manager_class.<locals>.JupytextContentsManager.get() got an unexpected keyword argument 'require_hash'

[mwouts]

Yes that's another item on my TODO list. Jupytext does not implement yet the require_hash argument (mwouts/jupytext#1165). With both Jupytext and Jupyter-collaboration installed I could save an .ipynb notebook, so I guess it's not as severe as the notebook corruption? Anyway let me have a look, maybe I can do something about that one too.

[mwouts]

I have a version with the require_hash argument (on top of the base64 fix) at

HATCH_BUILD_HOOKS_ENABLE=true pip install git+https://github.com/mwouts/jupytext.git@implement_require_hash

May I get a quick review on mwouts/jupytext#1202? I initially had a reservation about adding the require_hash argument when the original contents manager does not have it, but that would be complex to do, maybe now (e.g. 9 months later) I can require jupyter-server>=2.10 and assume that every base CM has the require_hash argument?

Also does the concatenation of the two hashes (inputs+outputs) look fine? Or should the hash have a certain length?

[davidbrochart]

I just tested your branch and it seems to work fine, but I admit I don't know how Jupytext works exactly, so I cannot say it's the right way to do it.

maybe now (e.g. 9 months later) I can require jupyter-server>=2.10 and assume that every base CM has the require_hash argument?

That sounds reasonable.

Also does the concatenation of the two hashes (inputs+outputs) look fine? Or should the hash have a certain length?

Yes I'm not sure you can combine hashes like that, since they are supposed to conform to an algorithm.

[krassowski]

Yes I'm not sure you can combine hashes like that, since they are supposed to conform to an algorithm

It does not matter in practice. All that frontend does is it compares last hash with hash on the server to decide if file changed on the server. Server can use any algorithm.

[krassowski]

More generally it looks like RTC and jupytext are competing for the get() and save() implementation (RTC redirects the traffic to y.js websocket on drive/content provider level, but jupytext expects the custom server-side content managers to be called). This means that at least pairing will not fully work when both are enabled even after sorting out the drive prefix and hash issues.
@mwouts @davidbrochart is this correct?

I am looking at the Contents Provider idea again and trying to understand if this could help.

[krassowski]

Or, are we all good because the RTC file loader can call the jupytext server-side ContentsManager.get to get the transformed notebook?

[krassowski]

Ok, I left some thoughts on jupyterlab/jupyterlab#16744 (comment)

[mwouts]

Thank you @davidbrochart and @krassowski for your feedback. I will release a new version of Jupytext with the above fixes soon (probably this Sunday).

Re a non contents manager implementation, Jupytext has the same read/reads etc methods as nbformat, but that's still in Python.

[davidbrochart]

Or, are we all good because the RTC file loader can call the jupytext server-side ContentsManager.get to get the transformed notebook?

Yes, from what I can see jupyter-collaboration uses JupytextContentsManager, so it should work nicely 👍
Thank you @mwouts!

[mwouts]

I have released jupytext==1.16.5 (on PyPI for now, conda-forge will follow). That version implements require_hash and solves the "base64" encoding issue. As a reminder that issue could also be addressed either in RTC or on the base contents manager, see here).

Hopefully that will get Jupytext and RTC working better together!!

Quick question @davidbrochart or @krassowski , when I look at the logs below I am wondering where the notebook is actually saved. Is it saved both to a classical .ipynb file and to the .jupyter_ystore.db file? When I rename my notebook I get some warnings as if the RTC extension was trying to still use the older "Untitled" name, do you see the same?

Logs of jupyter lab

[I 2024-12-15 19:58:59.420 YDocExtension] Watching file: test_rtc/Untitled.ipynb
[I 2024-12-15 19:58:59.422 ServerApp] Initializing room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b
[I 2024-12-15 19:58:59.468 ServerApp] Content in room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b loaded from file test_rtc/Untitled.ipynb
[I 2024-12-15 19:58:59.645 ServerApp] Kernel started: 6155df73-b91a-430c-96d0-56c2d282913c
[I 2024-12-15 19:59:00.316 ServerApp] Connecting to kernel 6155df73-b91a-430c-96d0-56c2d282913c.
[I 2024-12-15 19:59:00.321 ServerApp] Connecting to kernel 6155df73-b91a-430c-96d0-56c2d282913c.
[I 2024-12-15 19:59:00.325 ServerApp] Connecting to kernel 6155df73-b91a-430c-96d0-56c2d282913c.
[W 2024-12-15 19:59:00.402 ServerApp] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x70cabedddf50>
[I 2024-12-15 19:59:01.390 ServerApp] Saving the content from room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b
[I 2024-12-15 19:59:01.391 YDocExtension] Saving file: test_rtc/Untitled.ipynb
[I 2024-12-15 19:59:01.394 ServerApp] Saving Untitled.ipynb
[I 2024-12-15 19:59:05.139 ServerApp] Saving the content from room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b
[I 2024-12-15 19:59:05.140 YDocExtension] Saving file: test_rtc/Untitled.ipynb
[I 2024-12-15 19:59:05.143 ServerApp] Saving Untitled.ipynb
[I 2024-12-15 19:59:18.483 ServerApp] Saving the content from room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b
[I 2024-12-15 19:59:18.486 YDocExtension] Saving file: test_rtc/Real notebook name.ipynb
[I 2024-12-15 19:59:18.489 ServerApp] Saving Real notebook name.ipynb
[W 2024-12-15 19:59:18.503 ServerApp] 404 GET /api/contents/test_rtc/Untitled.ipynb?content=0&hash=0&1734292758500 ([email protected]) 1.16ms referer=http://localhost:8888/lab/tree/RTC%3Atest_rtc/Real%20notebook%20name.ipynb
[W 2024-12-15 19:59:18.503 ServerApp] 404 GET /api/contents/test_rtc/Untitled.ipynb?content=0&hash=0&1734292758500 (127.0.0.1): file or directory does not exist: 'test_rtc/Untitled.ipynb'
[I 2024-12-15 19:59:22.454 ServerApp] Saving file at /test_rtc/Real notebook name.ipynb
[I 2024-12-15 19:59:22.456 ServerApp] Saving Real notebook name.ipynb
[I 2024-12-15 19:59:22.495 ServerApp] Out-of-band changes. Overwriting the content in room json:notebook:202663f6-251c-426f-87ea-261e4d649b3b
[W 2024-12-15 19:59:22.520 ServerApp] 404 GET /api/contents/test_rtc/Untitled.ipynb?content=0&hash=0&1734292762516 ([email protected]) 1.04ms referer=http://localhost:8888/lab/tree/RTC%3Atest_rtc/Real%20notebook%20name.ipynb
[W 2024-12-15 19:59:22.520 ServerApp] 404 GET /api/contents/test_rtc/Untitled.ipynb?content=0&hash=0&1734292762516 (127.0.0.1): file or directory does not exist: 'test_rtc/Untitled.ipynb'

[goerz]

Just tested the new jupytext release, and it seems to work!

That's pretty exciting! Jupytext and the Collaboration plugin should complement each other very well, and it's also going to be extremely useful with the jupytext.nvim plugin that I've been rewriting during the last couple of days.

[davidbrochart]

Is it saved both to a classical .ipynb file and to the .jupyter_ystore.db file?

Yes, the notebook source is saved in the .ipynb file, and the Y updates are saved to the .jupyter_ystore.db SQLite file, by default.

When I rename my notebook I get some warnings as if the RTC extension was trying to still use the older "Untitled" name, do you see the same?

Are you renaming the notebook out-of-band e.g. from the terminal, or through e.g. JupyterLab? In the latter case, it should be supported, but in the former case, I'm not sure jupyter-server-fileid watches the file system, which would explain the warnings.

[krassowski]

I'm not sure jupyter-server-fileid watches the file system, which would explain the warnings.

Not yet - jupyter-server/jupyter_server_fileid#80

[mwouts]

Are you renaming the notebook out-of-band e.g. from the terminal, or through e.g. JupyterLab? In the latter case, it should be supported, but in the former case, I'm not sure jupyter-server-fileid watches the file system, which would explain the warnings.

Thanks! Well I was renaming the notebook from within JupyterLab - but maybe we should follow-up on that in another issue, when time permits I will see if I can reproduce and share more details.