Add support for associating a notebook with a project (#r project)

[jonsequitur]

The question comes up occasionally about whether a project can be referenced from a notebook. The use cases vary, so a discussion would be useful to understand the kinds of things people might want to use such a mechanism for.

Example scenarios:

• Initializing a notebook's NuGet dependencies
• Defining methods and classes that can be used from within the notebook

Some questions:

• Is it desirable to have an "automatic" association of some kind so that code from the project is invoked automatically based on some convention (e.g. the presence of a project with the same name in the notebook's folder)?
• What's the workflow for seeing changes in the associated project reflected back in the notebook?
• What kinds of projects should be supported?

[onionhammer]

Personally I see this feature being useful for the following:

1. I have a large domain project (csproj) that isnt a nuget package, but I want to reference it to generate stats, charts, reports etc from the domain (which accesses a database).
2. Building code that can be shared by multiple notebooks.

It would be great if an active notebook reloaded references when the project was rebuilt, but I think the user could just manually re-run the referencing cell as well

[thelazydogsback]

Also was looking for this so I could experiment with some of our internal libs along side of projects and provide "live documentation" for them.

It would also be great to see these notebooks integrated with the UnitTest runner, so I could interactively write testing code and turn them into unit tests by having cells do Asserts/Expects.

[jonsequitur]

Related: dotnet/roslyn#51480

[hypdeb]

Agreeing with previous comments : this feature would be a great way of creating interactive demos for library code.

[onionhammer]

I really think this would turn this project from a 'toy' with limited usefulness into an integral tool in my development toolkit.

With code reuse being so limited today it just isn't there. I would legitimately have .dib files in every C# project if this were a thing.

[funkrider]

I think that this feature would be great for REPL style workflows when developing. For example in a large project it isn't always possible or desirable to start the entire app and get the "State" of the app to where you want it to be and to start experimenting with code that manipulates the state. A notebook would be perfect for setting up a "state" of the app / lib / proj etc. and then being able to try out code in a self-contained way. To answer the initial questions here is my response:

• Is it desirable to have an "automatic" association of some kind so that code from the project is invoked automatically based on some convention (e.g. the presence of a project with the same name in the notebook's folder)? - Yes it would be awesome to be able to create a "notebook" from a project, for example a web API, that had auto generated cells that a) Referenced all the dlls used and b) loaded the default / environment config and could get the app running with some kind of initial "State".

• What's the workflow for seeing changes in the associated project reflected back in the notebook? Be great if "hot-reload" worked in your notebook. Say you start up your notebook, it has some initial "state" from config etc. then you execute a function, for example load up a List into a var in your notebook. You want to write code that manipulates your loaded list, but it doesn't have an "included" foreign key, say List, so you update the code that loads the customers in the project to also get their order, and hot-reload your notebook. When you run the cell to load your List again voila your list contains your child objects. You go on with your code that experiments with your desired List.

• Also would be awesome to be able to "bind" a function two-way to a notebook cell. Click a button that generates a new notebook with the config, refs etc. and the original source for your function is in a cell of your notebook. You manipulate the function in your cell and when you are happy - push it back to the project and update the original.

• What kinds of projects should be supported? I suppose anything that can run without a GUI would be a great start. Might be tricky with WPF interactions etc. Maybe having a way to "jack-in" to a running process would be great also. Say start up your Xamarin Forms project / app and then have your notebook be able to attach to the running process and execute code and retrieve returned values / state from the process.

Just as a side-note I did a bunch of Clojure work and the work-flow is heavily geared to a REPL. Being able to modify any function or state in a running process and being able to return / visualize the state in a meaningful way is development GOLD! For .Net folks it is hard to imagine how much more productive a great REPL experience can make you. .Net 6 hot-reload is a step in the right direction however returning and visualizing state is not quite there yet. Hot-reload is also "all" saved changes in the dll however with a REPL you can "push" just a single desired code update or state change, that would be nice too ;-)

[tthiery]

My use cases

• Developing a domain logic and presenting its capability and usage guidance to ...
  • interested parties during presentations
  • playable developer documentation
  • experimentation
• Data driven unit test development: I create a notebook, create some data there, throw it in an algorithm, see the outcome. Now I invoke a method to persist the triplet of input, algorithm and output into a data set which then extends the data set for data driven testing. Basically, while exploring a bug or developing the feature creating a test data set.

[lwestfall]

My use case:
Something I do fairly often when building proofs of concept is create a simple console application to drive whatever class library or other assembly I might be working on. In that case I'd have a project reference in the console app's .csproj to the other assembly. This is great because I can switch back and forth very quickly, intellisense / omnisharp are aware of my changes almost instantly without having to rebuild and restore, and when I dotnet run or debug my console app it will automatically rebuild and restore (if needed). It's also nice being able to (eg) "Go To Definition" from the console app.

So I guess what I'd love to see are all of those things, so that I could replace the console app driver in my proofing phase of development with interactive notebooks. These are the only big drawbacks to the current state of interactive notebooks in my opinion, for my specific use case. So to wrap it up neatly:

• Automatic rebuild/restore of referenced projects when changes are made.
• (near) Real-time analysis for intellisense including code changes that haven't been rebuilt
• Enable Go To Definition/Implementation etc. from within the notebook

In terms of what types of projects, I think C# class libraries are at the top of my list, personally.

[kMutagene]

This would not only help evaluating classlib projects in general, but also greatly improve the experience of authoring .NET interactive extension libraries, as testing them would be way easier. Currently, i have to package a project that exposes a .NET Interactive Extension as nuget package and reference it via #r "nuget: path" To test it locally without publishing to nuget. With project references in place, one could directly reference the project and iterate on the extension very fast.

In terms of what projects should be supported, F# classlibs are on top of my list. This would greatly improve my development workflow, which is currently referencing the .fs files in a .fsx script and testing library functions there. With project referencing from notebooks, i would use notebooks in all my projects for this type of quick testing and iterating.

[dcuccia]

+1 for living, testable/debuggable documentation with Notebooks, and to a fast inner-loop by using Roslyn to dynamically compile the notebook in-memory based on project source code changes. Right now, I have to close Notebooks so I can build my .dlls, re-open Notebooks to continue work. Combine with not getting the automatic transient dependency loading you'd expect, and the workflow gets quite tedious. Being able to compile and run cells alongside my integration tests would be next level.

[Peluko]

+1 I've just landed here searching for the exact same solution.

Edit:

The most simple thing that will work for me is automatic (hot?) reloading of referenced dlls. So I can keep a dotnet watch build on the project and #r the output dll.
If I do this now, I have to restart the notebook's kernel in order to get the dll version that just has been rebuilt, breaking the normal interactive development flow, since after kernel restart I have to rerun all the cells in order to get the notebook to the state where it was before restarting.

[onionhammer]

Is this progressing or scheduled?

[jonsequitur]

There is work happening on this front. There are a lot of great suggestions in this thread so we're trying to focus on what would be the most broadly useful and achievable features.

Thanks for the suggestions and keep them coming.

[dcuccia]

Hooray! The lack of Notebooks discussion at BUILD had me worried that the project wasn't moving forward. Can't wait to see how this gets shaped. For me, it could really be the next level of living documentation for our APIs—in our projects, in our GitHub README.md(s), on our websites...

[robconery]

@jonsequitur I am more than happy to help on this as needed. I'll ping you internally about it - this would be more than useful for a project I'm working on... in fact it would be a cornerstone!

[onionhammer]

Hopefully notebooks arent completely ignored come november .NET 7 release goodness...

[PawelStadnicki]

I was initially skeptical about referencing the project in a notebook and experimented only with plain NuGet references.
But I think I have found a use case that is blocked now and it may be solved via the project reference in the notebook cell.

I mean the F# -> C# interop here.
F# has type providers and they can be exposed somehow to C# via separated project/package.

Many type providers are based on static parameters that are resolved in a compilation phase.
So when I create F# package and expose it to C# in .NET Interactive, a lot of parametrization possibilities are gone.

Having the possibility to create a type from a project reference on notebook cell run will unblock it and enable huge F#/C# metaprogramming capabilities/interop.

So can the below be possible to do?

F# cell:
#r "project:projectname"

C# cell:
#r "nuget:projectname"

In particular it could enable everything I do with Fable-> F# to C# as below:

It could increase the target/audience by > 10x and be a nice tool to compare/mix/learn the languages

[onionhammer]

Skeptical? Interesting.. I could see adding notebooks to basically every C# project to demonstrate certain features of a library or make interactive utilities.

[PawelStadnicki]

I was initially skeptical about referencing the project in a notebook

I was initially skeptical about referencing the project in a notebook in comparison to just nugets, not about pairing notebooks with projects.

I'm one of the biggest advocates for using notebooks next to projects and even creating full solutions that are hosted on notebooks & .NET Interactive

[PawelStadnicki]

I simulated the project reference functionality in a cell with CliWrap that runs dotnet build/pack for a project located in the current directory.
It generates a new nuget package on each cell run with an autoincremented version number.
In the next cell it loads that nuget however it is always the same first version of the package.

I know it is by design and the same for every package even if I provide the version explicitly:
.

@jonsequitur I know it can be solved by restarting kernel but in my scenario it is not an option. I saw somewhere on github issue that you were thinking about using another AssemblyLoadContex, resigning because it is hard to reason about. But could it be a special case (option ) for those who want it deliberately?

Remark: you may ask what is the reason I need a new package version despite nothing having changed in the notebook or even in the underlying project. My project uses F# Type Provider that is based on string literal that is based on some unrelated file content that my projects uses under the cover ( it is complicated but it enables sophisticated metaprogramming)

[musicalmathmind]

Adding a +1 to this. My use case is that I'm integrating a 3rd party API. I have some DTOS in my project that help me interact with the API. Would love to be able to quickly prototype an API service class in the notebook, then transfer that over to my project. This would be transformative in the way I could explore integrating other services with my codebase.

[pederlh]

One of the things I miss when working with Polyglot notebooks is the possibility of using the 'Go to' commands (such as Go to definition, Go to references and Go to implementations).

This would make it much more practical to use notebooks as documentation. One specific use case is having notebooks that presents high-level functionality in a project, while making it easy to explore details in the relevant source files.

[Awsmolak]

+1 here too!

[dcuccia]

Just want to chime in again on this - as someone currently working to publish Notebooks as documentation, it would be so much nicer to be able to treat them as runnable and debuggable first class citizens of a project. Instead, I have to create and maintain static C# scripts in my project (so I can debug and test them with CI), and then copy-paste portions of that code into a parallel set of .ipynb notebooks that are maintained elsewhere. It would be so much nicer and less error prone to be able to run and test them directly.

[jonsequitur]

In case it's useful, notebook automation such as directly running and testing notebooks is supported by dotnet-repl.

This doesn't address the need to copy and paste code between the notebook and your project. The ability to reference project code (using classic C#, not C# script) within runnable documentation was implemented in dotnet try and is being updated and brought into .NET Interactive via CSharpProjectKernel as part of the Try .NET update currently in progress. This should eventually allow dotnet try-style Markdown or other formats to be editable and runnable in Polyglot Notebooks or other frontends.

[dcuccia]

Super useful, thanks @jonsequitur! I had no idea dotnet-repl existed, well done! Great news about the Try .NET update as well - glad to hear MS is making these important investments! Bright future for living/executable documentation. :)

[mahomedalid]

I agree with @dcuccia I think this would be perfect for the documentation of https://github.com/mahomedalid/dotnet-llm-eval-samples as most of ML people are familiar with jupyter notebooks.

[coldacid]

I hope to see something on these grounds in the near future. I have multi-project solutions and have scenarios where I need to mix code from two different projects that don't have a shared dependent but do have shared dependencies, and having to list out not just the two projects I'm pulling into a notebook, but also their own dependencies (both other projects and NuGet packages) leads to a kind of DLL hell. It's not uncommon for me to end up with a FileLoadException in a cell later down in a notebook because the kernel tried to load the same assembly from a different directory because of this forked project dependency path.

[zlsmith86]

I see this is no longer available in the visual studio marketplace. Are there plans to release it again?

[jonsequitur]

@zlsmith86 It's here:

https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-interactive-vscode