Align with fsharp.compiler service

[KevinRansom]

The FSharp compiler service project https://github.com/fsharp/FSharp.Compiler.Service has a lot of functionality that we can use in the visual fsharp tools language service. If we have this codebase in visual fsharp tools then we won't have to load the project into two different language services for vs's that have visual fsharp tools and the Visual FSharp Powertools.

@dsyme, @tpetricek, what do we need to do to make this happen?

Kevin

[forki]

I assume there are already lot's of additions in FCS which are not under CLA. It will be hard to make code transisiton work with Microsoft's current policy. But FCS come with Apache 2.0 and is on NuGet. So one way could be to trust the FCS project and just use the NuGet package. (I assume that's not what you want)

For the code merge we would have to ask 64 contributors (maybe a bit less) to sign CLA and to allow the merge. I assume most will allow it (at least when they are able to sign CLA).

If you consider to merge the projects then look at fsprojects/FAKE#598 - here @nathanaeljones prepared the license change for FAKE which worked very well.

[tpetricek]

I wonder if it would make sense to "go the other way" and build the Visual F# tooling on top of the F# Compiler Service? This obviously sounds like a lot of work, but it might actually be easier than trying to reconcile the changes in the F# compiler service with the main F# codebase.

Also, since F# PowerTools is also based on the F# Compiler Service, going this way might allow easier integration of PowerTools into the core Visual F# tooling.

That said, I can see that both ways are doable. It is probably a matter of what is the most doable option given the resources we have. I might be able to do a little bit of work on this with @dsyme when I'm back in Cambridge. How is this aligned with the future plans of the F# team at MSFT?

[latkin]

I think the compiler service codebase should really live with the compiler codebase, so that that stay aligned by default. New features/fixes in the compiler and language currently have lag time before they are reflected/supported by the service.

The first step is integrating FCS to VF#, and shipping it as part of VF# - this keeps it automatically aligned with the compiler codebase, and allows 3rd party tools to rely on it being a) present and b) up-to-date. From a purely technical standpoint I assume this won't be terribly difficult - just need to integrate the code/tests and find the right way to package it in VF#.

Migrating the VF# VS tools to use FCS to drive the in-box language service, etc is a much harder challenge, and will take longer. There are significant performance and quality obstacles that need to be resolved first.

[forki]

Please don't forget that FCS has a completely different release cycle and our whole ecosystem (at least the stuff in projectscaffold but probably much more) relies on fast patches to it. The delay from visual f# => FCS is currently a very small issue.

[ReedCopsey]

@forki It's possible that, should FCS merge into Visual F# repo, the fsharp/fsharp repository could still handle the quicker release cycles (as it'd then have a copy), with FSSF doing the nuget packaging as needed by the community. The biggest hurdle would then be making sure that commits to the compiler service were moved across to the other repo quickly, and the deployment was managed properly from there.

[dungpa]

As an early user of FCS, I'm in favor of keeping it an independent repo unless benefits of merging to Microsoft/visualfsharp hugely outweigh the costs.

FCS has been up for 15 months and there has been an ecosystem of community tooling springing around it. It's important to have a quick/flexible release cycle for FCS so that we have chances to screw things up and iterate. One may think that the patches to FCS are relatively small, but the repo has done pretty good jobs given that it has no dedicated maintainer.

[latkin]

Yes, the expected rate of contribution is definitely something that needs to be considered. Anything hosted under the Microsoft umbrella will inevitably move a bit slower, for various reasons. We need to balance that against the benefits of having stuff in the box. All else being equal, it just feels to me that in principle compiler services should be based on the same codebase as the compiler.

The lag becomes more of an issue if/when the VF# tools depend on FCS. If a new language feature is added, the VS tools need to work properly at that time, not later on when FCS hits a release cycle and has had time to merge the latest compiler code.

[KevinRansom]

Well, as it stands right now, Visual F# Tools and Visual F# power Tools each use a different language service to do their work, that's twice the bugs, twice the working set, twice the caching and twice the parsing. We believe we need to provide in the Visual F# Tools a single set of APIs to eliminate that duplication.

Moving forward we expect there to be a cross platform language service that the Visual F# Tools and perhaps the Visual F# Powertools are built on, so that we can operate on non Windows platforms, in our minds we expect the FCS to form the basis for that language service. The FCS provides that today, except that the Visual F# tools are not built on it.

As for the licensing I discussed this with the lawyers back in July last year, about the time I first raised this. Because fcs is a genuinely open source project with a variety of contributors that is released with the Apache 2.0 license we believe will be able to accept contributions from it. Of course we will reconfirm with them when we get the PR's but I don't believe it will be an issue.

@forki We do not expect to impact FCS release cycles after all, all they have to do is suck the code down and build a nuget package and they are done.

We believe that we need to do this in order to ensure that there is a high-quality, performant substrate for building F# experiences that is shared by all tool builders on all platforms. I know that it is work and a diversion of effort for some, but I am convinced the F# eco-system will find it advantageous for the FCS and Visual F# tools to be closely aligned.

[forki]

"a bit slower" ;-)
On Mar 27, 2015 7:58 PM, "Lincoln Atkinson" [email protected]
wrote:

Yes, the expected rate of contribution is definitely something that needs
to be considered. Anything hosted under the Microsoft umbrella will
inevitably move a bit slower, for various reasons. We need to balance that
against the benefits of having stuff in the box. All else being equal, it
just feels to me that in principle compiler services should be based on the
same codebase as the compiler.

The lag becomes more of an issue if/when the VF# tools depend on FCS. If a
new language feature is added, the VS tools need to work properly at that
time, not later on when FCS hits a release cycle and has had time to merge
the latest compiler code.

—
Reply to this email directly or view it on GitHub
#334 (comment)
.

[latkin]

Visual F# Tools and Visual F# power Tools each use a different language service to do their work, that's twice the bugs, twice the working set, twice the caching and twice the parsing. We believe we need to provide in the Visual F# Tools a single set of APIs to eliminate that duplication.

This isn't going to change by us merging/shipping FCS, or even using it for our own language service. As long an VFPT still exists and needs to run a language service, well, that's another language service.

do not expect to impact FCS release cycles after all, all they have to do is suck the code down and build a nuget package and they are done.

The point is that they want to add to the code not just consume it. And that there is a (justified) worry that flow of contributions will be slower if Microsoft holds the keys to the codebase.

[forki]

@latkin exactly. Currently we see situations where an issue is found and literally one hour later we have a new FCS version. After a couple of "paket update" calls it's distributed through the whole ecosystem.
FCS is REALLY important.

[ReedCopsey]

@forki While I think that's awesome, I do think, realistically, a (relatively small) delay would still be worth the benefits of consistency in the codebase, IMO. As long as the release can still be pulled out quickly once the PR is accepted, it seems okay to me.

[latkin]

Beyond consistency, it paves the way for improved tools from Microsoft, in VS/Windows and cross-platform, as Kevin mentioned. This is really important -- what is going to be of more long-term benefit to the F# ecosystem: no/weak Microsoft tools but 1hr turnaround on community projects, or strong/supported F# tools from Microsoft with reasonably slower turnaround?

[dungpa]

I think Lincoln has summed up the problems nicely.

@KevinRansom @latkin What's F# team's plan to improve FCS? There is a large number of issues/feature requests pending in FCS repo. I really hope with the change/alignment, quality of FCS will be improved, slowly but steadily.

[forki]

Don't get me wrong: MS investment in F# is super important. I hope they increase it to the max (especially with all the azure stuff going on). But there is a big ecosystem outside of MS. FCS became important because lots of other tools heavily relied on it and especially on the promise that we as a community can fix it "easily". We need to keep that promise. If we do this inside of this repo with a special "out of band" release schedule then everything will be fine.

[enricosada]

TL;DR Useless rush to merge. Choose a shared long term roadmap. For me a possible solution is make a common fcs component (server) and a shared vs.extension as wrapper (like roslyn).
f#tools and powertools are client. User can choose to upgrade what want, by default is ms tested version.
Compatibility by major/minor version.

MS open sourced repo changed a bit the status of open source fsharp flow.
Now all actors are opensource in github.
Another big component is nuget.
FCS is core of fsc/fsi/vs.net/xam/etc and really amazing but without community f# is not going anywhere (like VB.net, supported yes but without future ihmo).

visualfsharp repo = fsc/fsi, win package, vs extension
fcs = is community fork of compiler (used as service, with many refactoring and good features)
openfsharp = unix/mac package, crossplat fix for libraries
vspowertools = fantastic (required ihmo) additional features to vs
fsharpbinding

so duplicated effort and confusion, but ok, is the beginning :D

i think there are separate issues:

1. fcs in same repo as fsharp compiler (so fsc is aligned to compiler and compiler improve faster)
2. vs.net tooling (align vs extension and fsc, not duplicated between vstools and vspowertools)
3. visualfsharp need a slower and stable loop than community

the best long term (ihmo) is a single codebase (that not mean a single repo) like roslyn, corefx

I think we can fix some problems with versioning (master/stable branch) and distinct component (and team)
But if we use only one repo (openfsharp/visualfsharp/fcs) for all component, these are too glued for versioning

my idea (medium term, after fsharp 4, possible if start now):

• use nuget package where possible
• openfsharp = fcs/fsc/fsi/core libraries (master branch (v4.1?) + branch v4.0 (stable) for microsoft)
• visualfsharp = vs.net extension + package

1. fcs/fsc/fsi should be in the same repo.
   use a sane versioning and a core team, so prerelease fast package (from master) and a stable branch used by visualfsharp tools.
   I think the repo should be the openfsharp.
2. change the vs.net extension.
   instead of embedd the fcs as assembly, make it a client to another extension (the shared fsc as process, like roslyn), so the user can choose to use latest version or use ms stable. Both f#tools and powertools call that process, compatibility with version number
3. ms f#tools default package is vs extesion + stable shared fcs extension
   fcs extension and powertools are separated packaged, so user can choose what want to do.

[dsyme]

closing old discussion (some alignment has been done, more is needed, but we can start new discussions for that)