Extend Jupytext to C# and F#

[mwouts]

C#, F# and Powershell Jupyter notebooks were announced in this post from the .NET blog. A preview of Powershell notebooks was then provided by @TylerLeonhardt in this post.

In november 2019 (Jupytext 1.3) we extended Jupytext to Powershell. Now I'd be curious to extend Jupytext to C# and F#, and then experiment with
a) turning documentation repositories like e.g. PowerShell-Docs or C#-Docs into live interactive Jupyter notebooks with MyBinder
b) editing/running the .ipynb notebook through a paired script in Visual Studio Code.

The repository dotnet/interactive seems to be a great ressource to start with. There, one can find

• an interactive binder service, see NotebooksOnBinder (followed by instructions on how to install the dotnet framework locally)
• as well as example notebooks in the three languages.

This thread may also be of interest to @dfinke who authored PowerShellNotebook, a collection of scripts that can convert notebooks to scripts, and text to notebooks.

[mwouts]

The language in the notebook kernels is called C# and F#. However the Markdown files seem to use, instead ```csharp and ```fsharp for code cells.

[dfinke]

Thanks, it is. Very interesting.

[mwouts]

So far, this is what we've got...

On the PowerShell side:

• Jupytext (>=1.3.0) supposedly works well with PowerShell notebooks. Run pip install jupytext, restart the notebook server, and you will have the option to pair the notebooks to text files, using the Jupytext entries in the File menu (Jupyter Notebook) or the Jupytext commands (Jupyter Lab).
• With Jupytext, Markdown files and .ps1 scripts can be opened as notebooks in Jupyter (right-click / open as Notebook in Jupyter Lab), and then paired to an .ipynb file in Jupyter.
• On the simple examples that I had tested, the scripts could also be executed interactively in VS Code. I'd be interested in knowing whether this also works on more complex scripts/notebooks.

@dfinke , @TylerLeonhardt, would you be interesting in giving a try to the above? Do you like the idea to store the notebooks as Markdown files? Or as .ps1 scripts that one can also execute interactively in VS Code?

On the C# / F# side:

• I just created a PR that adds the support for the two languages.
• The Markdown representation seems OK, but at least the C# script does not seem right. The C# extension in VS Code underlines a few issues. Maybe we should wrap the notebook instructions in a namespace / class, as in the introductory example by w3schools. Also, the (LaTeXString) and (MathString) examples may be some kind of magic commands, maybe they should be commented out:
  

[dfinke]

I like storing it as markdown because I can use them as instructions on a repo and then the markdown can be re-purposed as an interactive notebook. Need to setup my env, I've only been running the notebooks in Azure Data Studio and the dotnet/interactive part with mybinder.

This is good stuff.

[mwouts]

Thank you @dfinke !

Installing Jupytext on mybinder should be easy (just pip install jupytext there). I am giving it a try at https://github.com/mwouts/interactive, and this is the link to the corresponding binder. There, I have converted all the example notebooks to .md files, and you will see that Jupyter Notebook opens them as notebooks, see e.g. this screenshot:

For the C# / F# side, who do you think I can ask advice to? Do you think @LadyNaggaga would be interested in helping here? What I need is 1) a better understanding at how to make the C# file, that represents the notebook, syntaxically correct and 2) information on whether a C# file can be executed interactively in VS Code, like the Python scripts with # %% markers.

[TylerLeonhardt]

@jonsequitur is also a good contact on the .NET Interactive side for C# and F#.

[jonsequitur]

Hello!

For Markdown support for C# and F#, you might take a look at what we've done in Try. NET. This experience is driven by Markdown-based code snippets: https://devblogs.microsoft.com/dotnet/creating-interactive-net-documentation/. The main difference between these approaches is that Try .NET currently runs full C#, meaning it needs namespaces, class definitions, etc. Try .NET hides that code from the user and provides context in the Markdown by annotating the code fences. It's more of a content authoring experience and not something we'd expect a notebook user to have to type, though.

The scripting dialect used in dotnet-interactive on the other hand allows top-level code. It looks like the compiler errors you're seeing are because VS Code (which gets language services from OmniSharp) doesn't understand the scripting dialect. We're working on language services equivalent to the Try .NET experience but for the scripting dialect, so that dotnet-interactive will be, among other things, a language server for C# and F#.

As for interactive execution, the dotnet-interactive APIs are available on nuget.org (https://www.nuget.org/packages/Microsoft.DotNet.Interactive/, https://www.nuget.org/packages/Microsoft.DotNet.Interactive.CSharp/1.0.0-beta.20103.7) but keep in mind these are early and Jupyter-agnostic. But we'd welcome any feedback and would be happy to help get .NET working in Jupytext.

[cartermp]

@mwouts perhaps some of the C# issues you're seeing here is because C# does not support top-level constructs and has various limitations with where you can place using declarations. This is something the C# language team is looking at addressing, since these kinds of ad-hoc inclusion of constructs is relevant for scripting and interactive programming.

For F#, if the resulting file is emitted as .fsx, then I'd expect things to mostly Just Work ™️, since F# supports the kinds of things I mentioned prior.

[mwouts]

Wow that's super interesting!

Thank you @cartermp for the advice on C# / F# scripts. I think I'll default the F# script extension to .fsx then, and leave the C# script as is, even if not syntaxically correct, it can still be useful if one want to copy/paste code from/to a notebook.

Thank you @jonsequitur for pointing at the dotnet/try project. It is great to see that you are using this on https://docs.microsoft.com, with e.g. the C# tutorial!

May I ask how the website itself (e.g. https://docs.microsoft.com/en-US/dotnet/csharp) is generated? Note that in the Jupyter community we tend to use Jupyter Book, , authored by @choldgraf , when we want to generate documentation sites from a collection of notebook - it may be interesting to compare both methods since I'm sure they face similar challenges.

Regarding dotnet/try, and its three flavors (web/local/Jupyter), I'd be curious to see if Jupytext can help mapping the .md files used by the web/local dotnet try to Jupyter notebooks. But I'm afraid we're not there yet... Indeed, when I look at the example mentioned in https://devblogs.microsoft.com/dotnet/creating-interactive-net-documentation/, i.e.

``` cs --region methods --source-file .\myapp\Program.cs --project .\myapp\myapp.csproj 
var name ="Rain";
Console.WriteLine($"Hello {name.ToUpper()}!");
```

I observe a few differences between try and jupytext's way to encode the cell metadata...

1. There is a space between ``` and the language
2. The language is cs, while I was thinking of using csharp in Jupytext. Don't you use more often csharp than cs in Markdown files?
3. Metadata is encoded as --key value, while Jupytext uses key="value" when value is a string.

How is the space important to you? Which of cs or csharp do you prefer? And for the metadata, maybe this is not really an issue - probably you won't use it in the notebook, and in theory, at least, Jupytext has a mechanism to preserve the metadata that it cannot parse...

[jonsequitur]

1. There is a space between ``` and the language
2. The language is cs, while I was thinking of using csharp in Jupytext. Don't you use more often csharp than cs in Markdown files?
3. Metadata is encoded as --key value, while Jupytext uses key="value" when value is a string.

All three of these are incidental in Try .NET because the command line parser isn't sensitive to whitespace, allows an arbitrary number of aliases (we support cs, csharp, and c#, and the capitalized forms of all three), and sees the following as equivalent: key value, key:value, key=value, key="value", etc.

The metadata should be ignored if not needed, though I also think it will be unimportant or at least different metadata for the scripting versus standard C# dialects.

[mwouts]

I see! I think I can implement something that is insensitive to the space between ``` and cs, and that preserves the Markdown file on round trips (except that space). See this test.

As the next step, maybe I'll try to add Jupyter C#+Jupytext+MyBinder to dotnet/try-samples, and see how well (or not...) the dotnet/try .md examples work when we open them as notebook...

[mwouts]

I have copied the DockerFile and NuGet.config from dotnet/interactive to mwouts/try-samples and added Jupytext with

# Install jupytext (dev version with initial support for C# and F#)
RUN pip install git+https://github.com/mwouts/jupytext.git@csharp_and_fsharp

Unfortunately that does not seem to work. There are two issues with
https://mybinder.org/v2/gh/mwouts/try-samples/master :

1. There are no .NET Jupyter kernels there (only Python 3 is available)
2. Jupytext is not installed (.md files don't have a Notebook icon)

Does anyone know what is wrong? Or maybe I should ask: what is the minimal addition to a repo that allows to run the notebooks there on MyBinder with the .NET kernels? Thanks!

[jonsequitur]

@colombod Any ideas?

[colombod]

I have added the jupytext line do our docker file and this is the outcome

Can't see any build log, did you notice any error when binder started creating your images?

[colombod]

And this is running it

[colombod]

On a side note why are you suing dotnet/try-sample repo for this?

[mwouts]

Hello @colombod , thank you for your help!

On a side note why are you suing dotnet/try-sample repo for this?

It's because I want to see how well (or not) the try-sample examples will work in Jupyter. The example are not .ipynb files, but Jupytext can convert them on the fly to (i.e. even open them as) Jupyter notebooks. Now the question is whether these notebook will run fine...

I did start with dotnet/interactive repo a few days ago. There, I

• added Jupytext to the DockerFile
• and converted the .ipynb files to .md.

That went well, i.e. I could still open/run these .md files interactively as notebooks (cf my screenshot above, or the binder link: https://mybinder.org/v2/gh/mwouts/interactive/master )

With the dotnet/try-sample it is a bit more difficult, for me at least, because there was no DockerFile there initially (and also, because I am an absolute beginner with Docker...)

Can't see any build log, did you notice any error when binder started creating your images?

I don't think so, but maybe I'll try to use docker locally and see how it works...

[jonsequitur]

A couple of Try .NET issues that might relate to Jupytext, as food for thought:

• support for publishing standalone Markdown, static HTML, WebAssembly hosts dotnet/try#214
• add support to dotnet try Markdown for specifying certain settings as file-wide defaults dotnet/try#187.

[mwouts]

Locally I can run the dotnet/try-sample examples, they work fine. But opening them with Jupytext does not really work, as... there's no code in the Markdown file!! So obviously the notebook is empty 😄

It is interesting to see how dotnet/try structures the document. The cell in Markdown file points at a subset --region of the code file indicated with --source-file. Here I have the .md file:

## Sum all numeric elements in a sequence

This sample uses `Sum` to get the total of the numbers in an array.

```cs --region sum-syntax --source-file ../src/AggregateOperators.cs --project ../src/Try101LinqSamples.csproj

```

and then an extract of ../src/AggregateOperators.cs:

using System;
using System.Collections.Generic;
using System.Linq;

namespace Try101LinqSamples
{
    public class AggregateOperators
    {
        (...)

        public int SumSyntax()
        {
            #region sum-syntax
            int[] numbers = { 5, 4, 1, 3, 9, 8, 6, 7, 2, 0 };

            double numSum = numbers.Sum();

            Console.WriteLine($"The sum of the numbers is {numSum}");
            #endregion
            return 0;
        }

[mwouts]

A couple of Try .NET issues that might relate to Jupytext, as food for thought:

• dotnet/try#214
• dotnet/try#187.

Interesting!

How will dotnet/try react if we duplicate the source in both the Markdown cell, and in the source file? (a bit like the %load script.py cell magic does in Python notebooks, which replaces the content of current cell with that of script.py)

Is it correct that dotnet try only reads the source file and ignore what is in the Markdown file? Maybe we could think of a) syncing in Jupytext the cell content from/to the source file and b) duplicating this information in the .md file. What do you think?

[jonsequitur]

How will dotnet/try react if we duplicate the source in both the Markdown cell, and in the source file?

Typically, it "replaces" the cell in the file with the cell in the editor, though it doesn't actually write the change to disk. The compilation happens in memory.

Is it correct that dotnet try only reads the source file and ignore what is in the Markdown file?

We support both directions.

[mwouts]

Thank you @jonsequitur . The dotnet/try Markdown files is another challenge, so I will see what we could do for that at #438 .

And for now I will ship the C# and F# support in Jupytext 1.3.4 (PR #429 ).
Thank you everyone for taking the time to answer my questions!