New code cells in F# default notebook are missing metadata, are treated as C#

[FH-Inway]

Describe the bug

I mainly write notebooks for F# and PowerShell, with the occasional C# cell in the mix. The default notebook language setting is set to fsharp.

The last few times I worked with the notebooks, I noticed that new code cells show in the top bottom corner my default language setting "fsharp - F# Script Code". However, if valid F# code is entered in the cell, it gets a lot of red squiggles (screenshot 1). When executing the cell, there are a lot of CS errors, indicating that the code is treated as C# (screenshot 2).

When I open the notebook in a text editor, I see that the metadata node of the cell is empty. Other working cells have metadata indicating F# in there (screenshot 3). See also a gist of the file in that state: https://gist.github.com/FH-Inway/34984cfacc39b27e206873048a16fb3e?short_path=51a4666

When I explicitely set the code cell via the language selector in the bottom right corner to F#, all is well again.

When I close a notebook with such an "invalid" cell, it is opened again right away, showing unsaved changes. The cell can now be executed successfully and after saving the file, the cell shows F# metadata.

Please complete the following:

Which version of .NET Interactive are you using? (In a notebook, run the #!about magic command. ):
Version: 1.0.522904+cdfa48b2ea1a27dfe0f545c42a34fd3ec7119074
Library version: 1.0.0-beta.24229.4+cdfa48b2ea1a27dfe0f545c42a34fd3ec7119074
Build date: 2024-05-05T09:37:02.5479468Z

• OS
  • Windows 11
• Browser
  • Edge
• Frontend
  • Visual Studio Code

Screenshots

Related issues

#3263 Polyglot Notebook: [DevExE2E][Regression] The kernelName and language show as csharp in the created Untitled-1.ipynb contents.
#3480 Default language not applied after open an python ipynb file
#2794 New cells use notebook default language instead of previous cell language

[soulemike]

Seeing same issue with PowerShell cells.

1. New polyglot notebook as PowerShell
2. In first code cell, already tagged as pwsh, run a command
3. Receive error The state of the current PowerShell instance is not valid for this operation.
4. Manually set the cell tag to pwsh, command runs as expected
5. Save file
6. Restart kernel
7. All cells that weren't manually tagged reset to C#

I downgraded to a few versions back to 1.0.4403* and saw similar results, though in the current version the save file retains the notebook level metadata, in those older versions even that would go away.

[FH-Inway]

Since this keeps bugging me, a few more observations:

Observation 1

• When a new notebook is created, it is created with an empty code cell as first cell. When the Default Notebook Language setting is set to fsharp, that cell shows "fsharp - F# Script" in the bottom right corner.
• If that notebook is saved, a file is created, but the open file keeps the "unsaved" indicator. The cell now shows "csharp - C# Script" in the bottom right corner.
• In the JSON of the file, the cell is listed with csharp in the metadata.
• valid F# code is interpreted as C# code

Observation 2

• Again, create a new notebook. This time, before saving, create a markdown cell before the first code cell.
• When that notebook is saved, no "unsaved" indicator is shown after file creation.
• The code cell still shows "fsharp - F# Script" in the bottom right corner.
• The JSON of the file shows the code cell with fsharp in the metadata.
• valid F# code is interpreted as F# code

Observation 3

• Same steps as observation 2
• then add a new code cell after the existing code cell; it is shown with "fsharp - F# Script" in the bottom right corner
• after saving the file, no "unsaved" indicator is shown
• JSON of the file shows the new code cell with "languageId": "polyglot-notebook" in the metadata
• valid F# code in the new code cell is interpreted as C# code

I could keep going with all kinds of weird results, some reproducible, some not (at one time, I had a markdown cell with csharp language in the metadata). It seems every variation of sequence when a notebook is saved, when and where a new cell is added, when the cell is executed, ... results in a different outcome. Though none of the outcomes so far result in the expected behavior, that a new code cell is created as a fsharp cell. This makes it an inconsistent and frustrating user experience.

I'm now on the following version of .NET Interactive:
Version: 1.0.526301+dd3fefb929d456fa10b0b9f7e0b2c65f6cfbee93
Library version: 1.0.0-beta.24263.1+dd3fefb929d456fa10b0b9f7e0b2c65f6cfbee93
Build date: 2024-06-22T11:52:32.3437483Z

Still Windows 11, Edge and VSCode.

[nhirschey]

This bug is because when you click "+ code" to create a cell, the "metadata" field inside the json is not populated, so the cell is parsed as C# by default.

Example text view of problem cell:

{
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "Error",
     "evalue": "(1,10): error CS1002: ; expected",
     "output_type": "error",
     "traceback": [
      "(1,10): error CS1002: ; expected"
     ]
    }
   ],
   "source": [
    "let y = 5"
   ]
  }

Add metadata and it works

If I manually re-select the "fsharp" language in the bottom right of the code sell, the metadata gets set and I can run the code using the F# kernel.

{
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "dotnet_interactive": {
     "language": "fsharp"
    },
    "polyglot_notebook": {
     "kernelName": "fsharp"
    }
   },
   "outputs": [],
   "source": [
    "let y = 15"
   ]
  }

[FH-Inway]

Thanks, that is the workaround I currently use. I hope we agree that this should not be the normal workflow for a user.

[FH-Inway]

@brettfo It seems you did the change with #2712 where a new notebook is initialized with the default language setting. Do you have pointers on how to get started with changing the behavior when new code cells are being added to a notebook? I've been looking at the code in vscode-common for a while, commands.ts in particular. But nothing obvious has been jumping out at me so far.

[jonsequitur]

Note that this affects only .ipynb files and not .dib files.

@DonJayamanne, does this ring any bells? It seems related to the removal of support for the custom metadata field and appeared around the time that was removed.

[FH-Inway]

@jonsequitur You Sir just made my day! Thanks for this much more practical workaround. Combined with the "Save notebook as..." command I can use .dib as editing format and .ipynb as publishing format.

[DonJayamanne]

@jonsequitur @colombod
Is it possible to debug the .net extension and let me know where the problem lies in .net interactive.
Also are you able to repro this at your end?
Jupyter and Julia extensions also adopted the change to drop custom metadata and didn't run into any issues.

[colombod]

It seems that the edit sent to the cell and notebook metadata is not persisted. I wonder if this is also part of the issue that opening a notebook marks it as modified

[colombod]

@jonsequitur @colombod Is it possible to debug the .net extension and let me know where the problem lies in .net interactive. Also are you able to repro this at your end? Jupyter and Julia extensions also adopted the change to drop custom metadata and didn't run into any issues.

If you step into this 👍

interactive/src/polyglot-notebooks-vscode-common/src/notebookControllers.ts

Line 84 in c384f80

const notebook = vscode.workspace.notebookDocuments.find(n => n.getCells().find(c => c.document === textDocument) !== undefined);

you see we calculate the metadata but after the edit is not persisted.

[DonJayamanne]

metadata but after the edit is not persisted.

Please can you share where this edit is made, thanks a lot

[arisliang]

it seems even i set the "Default notebook language" to be fsharp, the new cell is still c#. That's really annoying to have to manually override every time a cell is created.

[colombod]

@DonJayamanne this is the "race" we reported a while back. We need to deep test and isolate where the issue is, so far everything points to internal implementations in the vs side, but still need to isolate this.