Markdown text with accents produces invalid IPYNB file

[drdarshan]

Jupytext newbie here, so apologies if this is a known issue.

I created a markdown file containing an accented character. When I attempted to open it in JupyterLab the notebook had garbled text, see:

I also looked at the IPYNB file and it seemed to have incorrect JSON:

  {
   "cell_type": "markdown",
   "id": "6f176b80",
   "metadata": {},
   "source": [
    "LS0tDQpqdXB5dGV4dDoNCiAgZm9ybWF0czogaXB5bmIsbWQ6bXlzdA0KICB0ZXh0X3JlcHJlc2Vu\n",
    "dGF0aW9uOg0KICAgIGV4dGVuc2lvbjogLm1kDQogICAgZm9ybWF0X25hbWU6IG15c3QNCiAgICBm\n",
    "b3JtYXRfdmVyc2lvbjogMC4xMw0KICAgIGp1cHl0ZXh0X3ZlcnNpb246IDEuMTEuNQ0Ka2VybmVs\n",
    "c3BlYzoNCiAgZGlzcGxheV9uYW1lOiBQeXRob24gMyAoaXB5a2VybmVsKQ0KICBsYW5ndWFnZTog\n",
q    "cHl0aG9uDQogIG5hbWU6IHB5dGhvbjMNCi0tLQ0KRXQgdm9pbOAhIA0K"
   ]
  q},

Part of what makes this a bit inconvenient is that Jupytext auto-saved the notebook and overwrote my original markdown (which contained a whole bunch of other text) with the garbled cell. Is it possible to work around this issue or to detect a failure during conversion? Thanks in advance.

[mwouts]

Hi @drdarshan , thank you for reporting this!

May I ask how you created the Markdown file (which editor)? My understanding of what happens here is that it uses another encoding than utf-8, and because of that it is corrupted when Jupytext opens it as utf-8.

Anyway the example is interesting. I see that other editors like gedit here are able to detect an issue with the file, we could probably implement the same check in Jupytext:

Part of what makes this a bit inconvenient is that Jupytext auto-saved the notebook and overwrote my original markdown

I understand! Well if you want to avoid this, you should deactivate the autosave in Jupyter. Jupytext saves the notebook only when asked by Jupyter - either when you do an explicit save or when Jupyter autosaves.

[drdarshan]

Hello @mwouts, thank you for the explanation and suggestion to turn off auto-save from Jupyter! I am using Emacs 26.3 on Windows.

One thing I observed is that the jupytext cli does complain about being unable to read the markdown file, it seems the silent conversion error only happens when opening the file directly from the Jupyter UI. In that respect, a warning similar to what you showed in GEdit would be awesome.

Thank you again!

[mwouts]

Oh yes indeed that was specific to Jupytext within Jupyter. It should be fixed now in version 1.13.5. Thanks for reporting this!