Full FSI support on .NET Core 2.0

[dbettin]

From #2400

Properly handle the model for how .dlls are laid out on disk with .NET Core, thus fixing #r
Establish a good working model for how to do the above
Prompt to re-load an FSI session when #r’ing an assembly that targets a higher target than what FSI was launched with?
Error message asking user to re-load FSI manually?
Try to re-target and re-reference everything on the fly?

[KevinRansom]

@dbettin it's like you are channeling my nightmares today. Assembly reference and loading in a coreclr world requires a lot of thought too. FSI was and to a lesser extent the desktop FSC were very helpful in finding and resolving assemblies ... at the price of a lot of determinism.

It's a goal we need to fix by the time .NetStandard 2.0 is released. The details however, are not yet available.

[dsyme]

My comment from the fsharp-opensource mailing list:

There’s been progress – we’ve had variations on this working – but there’s a strong feeling that we need to get “#r on a nuget package” designed, implemented and finalized before the experience will be correct and stable for both .NET Core and .NET Framework. See also #2483

[KevinRansom]

Support for FSI on dotnet core.

Yes it will happen absolutely. That it hasn't yet, is extremely embarrassing to me. The reason is technical, rather than priorities or anything else. Well priorities have been impactful ... but they are not the main reason.

The issue is this: we test fsi.exe on coreclr and have done for a long ... long time, and it works well. Unfortunately ... (and this is the embarrassing bit) it relies on a bug in the coreclr to work ... in AssemblyLoadContext. If we fix the code to what we need it to be now that the bug is fixed in the coreclr, the entire way we test the coreclr build needs changing. This is the exact same reason why we still rely on project.json in our build.

I fully understand the irony of what I just said above ... if we fix the bug we can't test it ... if we don't fix the bug we can't ship it. I hate myself for this ... and if you want to expose me to the universe for the full idiocy of the position I just expressed I know how insane it is.

We must fix our build ... we must fix the testing ... I promise it will happen ...

I actually prefer Don's explanation, because coreclr FSI without built-in package management is definitely going to require external packagement to make scripts that reference libraries reliable.

However, in reality releasing fsi is not gated on adding packagemanagement it is gated on us digging out of a historic build/test mess that I created for us.

I am truly sorry about that.

Kevin

[forki]

What annoys me personally is that we bundle both things. We could already have a nice package management story in classic fsi and could make HUGE progress on FAKE 5. But we wait for netcore.... Am 01.09.2017 9:52 nachm. schrieb "Kevin Ransom (msft)" < [email protected]>:

…

Support for FSI on dotnet core. Yes it will happen absolutely. That it hasn't yet, is extremely embarrassing to me. The reason is technical, rather than priorities or anything else. Well priorities have been impactful ... but they are not the main reason. The issue is this: we test fsi.exe on coreclr and have done for a long ... long time, and it works well. Unfortunately ... (and this is the embarrassing bit) it relies on a bug in the coreclr to work ... in AssemblyLoadContext. If we fix the code to what we need it to be now that the bug is fixed in the coreclr, the entire way we test the coreclr build needs changing. This is the exact same reason why we still rely on project.json in our build. I fully understand the irony of what I just said above ... if we fix the bug we can't test it ... if we don't fix the bug we can't ship it. I hate myself for this ... and if you want to expose me to the universe for the full idiocy of the position I just expressed I know how insane it is. We must fix our build ... we must fix the testing ... I promise it will happen ... I actually prefer Don's explanation, because coreclr FSI without built-in package management is definitely going to require external packagement to make scripts that reference libraries reliable. However, in reality releasing fsi is not gated on adding packagemanagement it is gated on us digging out of a historic build/test mess that I created for us. I am truly sorry about that. Kevin — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2407 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNE-0l8H-ykoQUvECZk7xge5hYyzaks5seGCagaJpZM4L90AM> .

[KevinRansom]

I haven’t bundled both things. I’m saying, and I think quite clearly, that my stupidity on the build and test is what is blocking this. Fixing the build and test will enable FSI. Good Package management in FSI will make the world pretty groovy. Kevin KCR: Edited to remove email quote.

[matthid]

@KevinRansom I just really wanted to write that I appreciate and value your response. While I can see that this is embarrassing it feels like honesty is really a good way forward for this repository.

Too often (and maybe even only for historical reasons) we feel like there is some hidden agenda or political reasons behind stuff. So this really helps a lot. Thank you.

About what @forki said: I think a response on the other issue regarding the latest suggestions from the community might help.

[dsyme]

Unfortunately .it relies on a bug in the coreclr to work ... in AssemblyLoadContext.

I wasn't aware of this. Do you have a link to the bug for this? Thanks. Also a link to the place in the F# code where we are relying on this. Full transparency please :)

[KevinRansom]

The bug is in how you implement load contexts. What we do in FSI results in a stack overflow on bug fixed coreclr’s. The fix looks like this: https://github.com/Microsoft/visualfsharp/pull/3140/files#diff-3229c72e3fbd323c1dde9692aceedd65L322 It was part of PR to basically enable FSI on netstandard 2.0. However, work you and I have done superseded this PR, apart from this little bit.

[matthid]

Just to add: The runtime people suggested to me to use my own assembly load context (as I was hitting some similar issues with FAKE at the time), since I have done that the code and the loading logic feel a lot "cleaner".

Technically, this also fixes a lot of outstanding problems in FAKE like that you cannot use a different fcs version in your script code (for example when using fsf).

So maybe this should be something to consider. But yeah especially for fcs this might be breaking as existing users might depend on having the same set of assemblies available in the script code as in the host code.

[halcwb]

Is there any progress regarding this issue?

[cartermp]

@halcwb Not at the moment, no. We've been occupied primarily with some critical issues in existing workflows for .NET Core and IDE support. Hoping to pick up the in progress work on this soon.

[ghost]

fsi.exe that comes with SDK 2.1.302(.dotnet/sdk/2.1.302/FSharp/fsi.exe) works, you get missing reference exceptions sometimes (e.g. System.IO.File), but it takes all of 1 minute to work around it:

Create a F# project and add the functions you want, e.g. let read = System.IO.File.ReadAllBytes, then compile this project, and #r the resulting DLL in fsi.exe

As a nice plus, it works well with VsCode, including colours. There are some readline warts, but its still worlds better than mono fsharpi

[cotyar]

@abk-x How you managed to run it please?
Cannot get neither .dotnet/sdk/2.1.302/FSharp/fsi.exe nor .dotnet/sdk/2.1.400/FSharp/fsi.exe working on Win10 x64

[zpodlovics]

Suprisingly it starts - at least on linux (ubuntu 16.04 x86_64) - however it comes without readline (~console navigation) like functionality so it's really painful to use it, you cannot even navigate within a line and edit it without deleting the earlier chars.

The assembly names are different in .NET Core (also there are ref only assemblies), you don't have to compile any additional dll you just have to found the correct name where the actual function / method implemented. For example the System.IO.dll is just looks too small to contain any real implementation.

-rw-r--r-- 1 root root  768512 Jul 25 18:27 System.IO.Compression.Native.so
-rw-r--r-- 1 root root   58880 Jul 25 18:26 System.IO.Compression.Brotli.dll
-rw-r--r-- 1 root root  258560 Jul 25 18:26 System.IO.Compression.dll
-rw-r--r-- 1 root root    4096 Jul 25 18:26 System.IO.Compression.FileSystem.dll
-rw-r--r-- 1 root root 2688244 Jul 25 18:26 System.IO.Compression.Native.a
-rw-r--r-- 1 root root   24064 Jul 25 18:26 System.IO.Compression.ZipFile.dll
-rw-r--r-- 1 root root    5120 Jul 25 18:26 System.IO.dll
-rw-r--r-- 1 root root   18432 Jul 25 18:26 System.IO.FileSystem.AccessControl.dll
-rw-r--r-- 1 root root  203264 Jul 25 18:26 System.IO.FileSystem.dll
-rw-r--r-- 1 root root   57856 Jul 25 18:26 System.IO.FileSystem.DriveInfo.dll
-rw-r--r-- 1 root root    4608 Jul 25 18:26 System.IO.FileSystem.Primitives.dll
-rw-r--r-- 1 root root   73728 Jul 25 18:26 System.IO.FileSystem.Watcher.dll
-rw-r--r-- 1 root root   68096 Jul 25 18:26 System.IO.IsolatedStorage.dll
-rw-r--r-- 1 root root   57344 Jul 25 18:26 System.IO.MemoryMappedFiles.dll
-rw-r--r-- 1 root root   13312 Jul 25 18:26 System.IO.Pipes.AccessControl.dll
-rw-r--r-- 1 root root  112640 Jul 25 18:26 System.IO.Pipes.dll
-rw-r--r-- 1 root root    4608 Jul 25 18:26 System.IO.UnmanagedMemoryStream.dll

You can take a look each assembly source code here:
https://github.com/dotnet/corefx/tree/master/src

Example usage:

Microsoft (R) F# Interactive version 10.2.3 for F# 4.5
Copyright (c) Microsoft Corporation. All Rights Reserved.

For help type #help;;

> #r "System.IO.FileSystem.dll";;

--> Referenced '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/System.IO.FileSystem.dll' (file may be locked by F# Interactive process)

> let s = System.IO.File.ReadAllBytes("/tmp/x");;
val s : byte [] = [|120uy; 10uy|]

[nosami]

You can use --readline+

dotnet /usr/local/share/dotnet/sdk/2.1.302/FSharp/fsi.exe --readline+

[zpodlovics]

@nosami Well, almost - you can turn it on, but right now the implementation is just unusable slow (the drawing rate is roughly ~0.5 char/sec - deleting a char from a long line will take more than a minute...). Probably disabled by default for the same reason.

[nosami]

@zpodlovics

Strange - it seems ok on OSX

[nosami]

@zpodlovics - I see what you mean now, there is definitely something wrong with readline when used from dotnet

Here I try editing a long line, first from mono's fsharpi, then using the .Net core version.

[zpodlovics]

The call flamegraph looks horrible (my earlier example session recorded with perf):
https://gist.github.com/zpodlovics/68eab3f5188313e33c665ed1a9af4c09

[zpodlovics]

I have managed to get a source build of fsi.exe (using the tricks from #5636) and created a quick callgraph profiling (using --readline+ option). The first 4/5 of the callgraph is F# startup, the more interesting parts is on the remaining 1/5.

On backspace the writeChar / render loop looks really weird as it use some deeply nested read/change call chain.

The svg callgraph are available here (download it if you want zoomable interface):

https://gist.github.com/zpodlovics/28c61d357b054bd4659cd5b8bd7294d3

A typical callstack looks like this from the callgraph:

The text profile are available for speedscope (https://github.com/jlfwong/speedscope or online https://www.speedscope.app/):

https://gist.github.com/zpodlovics/d0ac67c47680df1e68c0704e918c2b16

[AviAvni]

@zpodlovics from the code https://github.com/Microsoft/visualfsharp/blob/master/src/fsharp/fsi/console.fs#L381-L470
it's look that something with the tail call is not working
did you profiled the fsi.exe from the Debug or Release directory?

[zpodlovics]

@AviAvni I have tried out originally with the SDK packaged fsi.exe which is a release build (and unusable when readline enabled), but I was not able to resolve the managed symbols in the profile. It was probably the debug version that I tested, but not 100% sure.

I did a new profile using the release build from source, it have the same shape:

The svg callgraph are available here (download it if you want zoomable interface):

fsi-netcore-callgraph-2018-09-16_2-svg.zip

The text profile are available for speedscope (https://github.com/jlfwong/speedscope or online https://www.speedscope.app/):

fsi-netcore-callgraph-2018-09-16_2-txt.zip

[AviAvni]

@zpodlovics the problem with --readline+ is also happen in the .net framework fsi on windows

[mtnrbq]

Is there a rough view on the ETA for this support ?
Am I correct that we missed (aka were thrown off) the v2 Azure Functions boat already? - Azure/azure-functions-host#3414

[cartermp]

We have the necessary infrastructure built in and a first run at a NuGet plugin built. @KevinRansom can share more details, but I would expect this to land in a .NET Core SDK update before a Visual Studio update.

(Note: syntax of #r is not finalized; this is just so there is a straightforward parser)

And yes, you are correct about Azure Functions. Neither us nor them wanted to have a hacked up version of FSI running on production for script-based functions. You can still use Functions by creating a library project and referencing their package(s), and it will all just work.

[costincaraivan]

Any news on this front? 😨

[cartermp]

Closing this as it's now in preview: https://devblogs.microsoft.com/dotnet/announcing-net-core-3-preview-3/

dotnet fsi --readline is what you can use now.