Paket support for vnext project files

[Stift]

With VS2015 and ASP.NET 5 a new project format for asp.net projects is introduced. This has the new extension .kproj and beside of that exists a project.json (https://github.com/aspnet/Home/wiki/Project.json-file) which describes the dependencies of that project.
The problem here is, that vNext comes with its own package manager (kpm), that downloads all dependencies by default in the folder user profile (subfolder: ".k"). I think that should keep global dependencies locally (per profile). KPM seems just to be enabled to operate with the nuget API.
This leads to the situation that in one solution we can have nuget and KPM side-by-side.

I don't see currently any solution how I can enable Paket with vNext when I switch some projects to vNext. I only see currently that you can specifiy which kre (K runtime) where maybe paket could be triggered from to resolve the references.

Has anyone else already investigated that issue or some ideas that we can try out?

Side note: the documentation of kpm is very basic and a lot of its commands are not documented.

[forki]

yep. that thing is pretty bad and we don' know how to get into this stuff, yet.

[Stift]

I currently try to find out how something can be hooked up in the KRE.

[forki]

OK, let's discuss what would be needed.

• So far I see a project.json file per WebApplication project. This file is json and could be easily edited from paket. We would need to write package name and resolved version (from our lock file) into this file.

    "dependencies": {
      "EntityFramework.Commands": "7.0.0-beta8",
      "EntityFramework.SqlServer": "7.0.0-beta8",
      "Microsoft.ApplicationInsights.AspNet": "1.0.0-beta8",
      "Microsoft.AspNet.Authentication.Cookies": "1.0.0-beta8",
      "Microsoft.AspNet.Authentication.Facebook": "1.0.0-beta8",
      "Microsoft.AspNet.Authentication.Google": "1.0.0-beta8",
      "Microsoft.AspNet.Authentication.MicrosoftAccount": "1.0.0-beta8",
      "Microsoft.AspNet.Authentication.Twitter": "1.0.0-beta8",
      "Microsoft.AspNet.Diagnostics": "1.0.0-beta8",
      "Microsoft.AspNet.Diagnostics.Entity": "7.0.0-beta8",
      "Microsoft.AspNet.Identity.EntityFramework": "3.0.0-beta8",
      "Microsoft.AspNet.IISPlatformHandler": "1.0.0-beta8",
      "Microsoft.AspNet.Mvc": "6.0.0-beta8",
      "Microsoft.AspNet.Mvc.TagHelpers": "6.0.0-beta8",
      "Microsoft.AspNet.Server.Kestrel": "1.0.0-beta8",
      "Microsoft.AspNet.StaticFiles": "1.0.0-beta8",
      "Microsoft.AspNet.Tooling.Razor": "1.0.0-beta8",
      "Microsoft.Framework.Configuration.Abstractions": "1.0.0-beta8",
      "Microsoft.Framework.Configuration.Json": "1.0.0-beta8",
      "Microsoft.Framework.Configuration.UserSecrets": "1.0.0-beta8",
      "Microsoft.Framework.Logging": "1.0.0-beta8",
      "Microsoft.Framework.Logging.Console": "1.0.0-beta8",
      "Microsoft.Framework.Logging.Debug" : "1.0.0-beta8",
      "Microsoft.VisualStudio.Web.BrowserLink.Loader": "14.0.0-beta8",
      "FAKE": "4.3.0"
    },
  

But this would reintroduce mergeconflicts outside of paket.lock ;-(
Ideally we would let this block empty and just use references file.

• There is a global.json file which doesn't contain a dependencies block
• there is no packages folder anymore ;-(
• If you save project.json then VS restores automatically. How can we disable that or make it use paket restore?

[forki]

/cc @ctaggart

[keichinger]

@forki I'm not sure how easy and good it is to leave the dependencies block empty as it's not just only required for Visual Studio and tools such as OmniSharp to know about specifiy dependencies, it is also required by the build tools to resolve build dependencies. Keep in mind that the .kproj files are a combination of packages.config and .csproj files. Additionally you can not only specify global dependencies for your project through the root dependencies block, but you can also do this per architecture for scenarios where a specific package is not (yet) ported to make it work cross-plat. This makes it necessary to reflect this kind of feature in paket.references files.

Another thing about dependencies: You're able to override NuGet packages with a local version by simply specifiying a path to the source code

Furthermore building the actual project would also require Paket to pre-process project.json before it can actually build. Though this could be done through the prebuild event, which could execute Paket which then modifies the project.json etc.

Maybe this one here helps clarify a few things: https://github.com/aspnet/Home/wiki/Project.json-file

[forki]

Thanks. I get the feeling that from a dependency "management" standpoint things even got worse. Bit hopefully it's just my lack of understanding right now.

[forki]

Keep in mind that the .kproj files are a combination of packages.config and .csproj files.

Exactly. Now things are even more coupled.

[Mpdreamz]

Related. I would love some to get some more voices on this issue aspnet/dnx#2332

[forki]

just to clarify my "merge conflic" comment in #736 (comment)

the issue is: project.json is basically a merge of packages.config and csproj (maybe with better format).
But since paket would need to generate the dependencies block (and the block contains version numbers) every package update needs to touch the project.json. So if two persons in the team update at the same time you will get a merge conflict there. At the worst part is: you can't gitignore that file since it also contains the rest of the csproj information.

[forki]

/cc @nosami @davidfowl @yishaigalatzer @csharpfritz do you see a solution for us to make this work?

[yishaigalatzer]

Not sure why you need paket un this world (not to say you need or not), I would start with describing the scenarios paket improves in this world and understand the value proposition, then dicuss how to fit it in

[isaacabraham]

@yishaigalatzer I think that some of the problems that paket is trying to solve is to remove project-level versioned dependencies, which are still present in project.json, as well as (apparently?) having to store all dependencies, not just top-level dependencies, at the project level. Have a quick read of the Paket FAQ - the first section answers this question pretty succinctly (http://fsprojects.github.io/Paket/faq.html#I-don-t-understand-why-I-need-Paket-to-manage-my-packages-Why-can-t-I-just-use-NuGet).

[Mpdreamz]

@yishaigalatzer I tried doing that on the previously linked DNX ticket:

• repeatable builds (something still not solved with dnu/project.json)
• greenfield project developer experience (by separating the act of specifying the dependencies and their resolved versions)
• mature command line tooling
• managing the dependencies of many projects at once
• better dependency resolution algorithms
• .... insert all the other reasons why you generally pick paket over nuget

[yishaigalatzer]

Most of these reasons above do not apply.

With project.json you

1. Only specify top level dependencies
2. Have a stable and fast resolution (and much better defined than nuget 2.x)
3. Ability to float versions without any changes to the project files
4. decoupling of the project system from the package manager, allowing for command line tools to work equivalently as in visual studio tooling

So although some things are not available (like solution level dependencies), I would challenge to define the needs and then how to deal with them.

[forki]

@yishaigalatzer we don't want to get rid of project.json. but we think that all the major issues why we created paket in the first place still apply. We wanted to discuss how we can integrate paket into DNX, CoreClr and whatever else might come. In the new spirit of "open microsoft" we thought you might want to join the discussion.

[Mpdreamz]

2. true, but I personally still disagree with resolving to lowest by default (Add nuget level option for "resolve highest" NuGet/Home#1192). Do not want to turn this into a discussion about which is best but the fact I can choose paket means I have the option to choose what works for me.
3. Please read the linked dnx ticket. floating versions with no lock file will break repeatable builds. This has me the most worried.
4. You say this but there is no real decoupling since project.json lists both dependencies as well as configs/frameworks/sources/compilationOptions/commands/etcetera.

Not designing the dependency story around being able to manage 2+ projects easily at once will hurt adoption as it will be a PITA for folks willing to give this newfangled .NET ecosystem a try.

[Mpdreamz]

In fact the only thing that is "solved" now is transitive dependencies. I put that in quotes because I do not think not having a lock file and relying on resolving to lowest is the most elegant way to fix that problem and so for me that would still be a reason to chose paket.

[yishaigalatzer]

You need to keep in mind that there are two project.json's.

One for ASP.net where it is the project file and the set of dependencies and another where it just describes the dependencies so the discussion needs to be separated.

The common thing between the two is the responsibility to write a project.lock.json file.

I would also like to address all the concerns you raised above, such that managing per solution is easily done. It not going to happen overnight but it will.

There is room for different opinions (like pick higher dependencies) I see the desire and understand it, even if I don't necessarily see eye to eye.

My teams goal is to make NuGet better every day hopefully good enough such that Paket is not a necessity to achieve your goals.

However that doesn't mean we can't help it work in this new world. I don't have all the answers at the moment, so lets keep this discussion going.

I still believe your first goal should be to produce a proper lock file and for now keep up with changes we are going to introduce until it is released as RTM.

[csharpfritz]

NuGet supports restoring with alternate dependency version choices. Check
the dependencyVersion configuration option in NuGet.config:
http://docs.nuget.org/Consume/NuGet-Config-Settings

It can also be specified as a switch -DependencyVersion when calling
Install-Package

On Mon, Nov 9, 2015 at 12:51 PM, Yishai Galatzer [email protected]
wrote:

You need to keep in mind that there are two project.json's.

One for ASP.net where it is the project file and the set of dependencies
and another where it just describes the dependencies so the discussion
needs to be separated.

The common thing between the two is the responsibility to write a
project.lock.json file.

I would also like to address all the concerns you raised above, such that
managing per solution is easily done. It not going to happen overnight but
it will.

There is room for different opinions (like pick higher dependencies) I see
the desire and understand it, even if I don't necessarily see eye to eye.

My teams goal is to make NuGet better every day hopefully good enough such
that Paket is not a necessity to achieve your goals.

However that doesn't mean we can't help it work in this new world. I don't
have all the answers at the moment, so lets keep this discussion going.

I still believe your first goal should be to produce a proper lock file
and for now keep up with changes we are going to introduce until it is
released as RTM.

—
Reply to this email directly or view it on GitHub
#736 (comment).

[isaacabraham]

@yishaigalatzer it's great that you're reaching out and want to help solve this. Your team's goal, however, should not be to make Nuget better so that "hopefully" Paket is not necessary. As you've rightly pointed out, Paket and Nuget have different goals that might never fully converge e.g. solution-wide vs project-wide version control.

There's enough room in the .NET world for a second package management tool, and vnext project files should have the ability to support both.

[davidfowl]

You can expect that there will be tons of overlap in the 2 solutions and paket was made to try to solve real issues that the f# community had with nuget. The fact is that we are trying to improve nuget (and it's happening). Both package managers may coexist but expect the uniqueness of paket to diminish over time. Like Yishai said, we may have different philosophies but in the end the problem domain is the same.

We should always look for ways to collaborate but we are trying to build something that solves the same problems.

[yishaigalatzer]

My team goal is to make NuGet good enough such that it is not necessary to fork the eco system. There is room for more than one package manager for sure.

I do believe that there shouldn't be a forcing need for another package manager, paket raised a lot of good concerns and it will be silly not to address them in the mainline scenarios for users.

It is not a goal to make paket go away, but it is a goal to excel and move as fast as we can in improving and introducing new features to address the community concerns.

[isaacabraham]

@yishaigalatzer OK. I was simply replying to your original comment, in which that is what you said.

I think "forking the eco system" is a bit extreme. Forking the eco-system to me would mean replacing NuGet packages with some other form of package, and hosting them somewhere else. Paket was never about that as far as I know because it was designed to be a drop-in replacement for NuGet tooling.

FWIW, like I said, it's great to have some discussion on this, and I'm sure that NuGet will continue to improve.

[Mpdreamz]

@yishaigalatzer @csharpfritz @davidfowl Thanks for taking the time to respond, really I do appreciate it, but I feel you're brushing off my comments too quickly and managing rather then addressing my issues.

Its great you guys want to obsolete paket but nothing in the current state screams that the same goals are a priority *at all. I can rehash the importance of locking by default, resolving to highest, separating listing dependencies and using dependencies & managing deps globally but really I feel like I've exhausted all the different ways I can explain this on this issue already

• There is a -DependencyVersion thats great and all but with no clear education on the importance of a lock file developers might switch it on in their nuget.config without checking in their lock file. With DNX they simply can't because it contains a LOT more then just the resolved dependency chain.
• The fact that there are two project.json's or more aptly put dnx supersetting the same file is NOT in anyway decoupling the package manager from the project system.
• Paket is not just solving f# programmers problems.

How up to date is: https://docs.nuget.org/consume/projectjson-format ? Some of the updates on locking discussed here aspnet/dnx#2332 (comment) are not reflected there it seems. Genuinely wondering I fully understand docs lacking while things are in flux :)

I would love if the new system was designed to accommodate both rather then one trying to obsoleting the other. It would make our ecosystem stronger in the long run.

To be frank, I don't even care if nuget converges with paket because paket already does what I want extremely well but it needs to have a fighting chance in the new world.

[Mpdreamz]

However that doesn't mean we can't help it work in this new world. I don't have all the answers at the ?
moment, so lets keep this discussion going.

Misread this line @yishaigalatzer, yes! So much this. Neither do I have the answers, but it would be awesome if we can create a story for Paket with dnx/coreclr/nuget's project.json before they hit GA 👍

[forki]

It's sad to hear that you still think we "forked the ecosystem". in every talk I tried to make very clear that this is not our goal. (see http://forki.github.io/FAKE.Intro/#/36). We always wanted to play well with the existing feeds and project systems. We even added features to work around VS bugs that is assuming nuget structure for test runners and so on. Yes we wanted to have different view on our dependencies and solve serious issues, but we didn't create new feeds or package formats or something.

[yishaigalatzer]

Given that this thread is about how to make Paket work with project.json scenarios, I'd like to abstain from derailing this thread any further. I'll be glad to answer/help with specific questions on how to do stuff.

[Mpdreamz]

@yishaigalatzer there have been several specific questions on here that have not been answered:

/cc @nosami @davidfowl @yishaigalatzer @csharpfritz do you see a solution for us to make this work?

This pertained to this comment: #736 (comment)

But it solicited a "why would you want this?" comment from you to which I offered several scenario's that Paket handles right now that NuGet does not. I am not all interested in a discussion on which one is better nor do I feel a need to bash NuGet.
Blanket responses that you guys are working hard and NuGet improvements are coming are not helpful here. As much as I really do appreciate all the hard work you guys are doing!

How up to date is: https://docs.nuget.org/consume/projectjson-format ? Some of the updates on locking > discussed here aspnet/dnx#2332 (comment)
are not reflected there it seems. Genuinely wondering I fully understand docs lacking while things are ?> in flux :)

Since DNX uses the project.lock.json for much more then dependencies this file is impossible to generate which is why I opened aspnet/dnx#2332 (comment) in the first place. There are plenty specific questions on that thread still unanswered too, nor a follow up on the new path being chosen. Is generating a resolved.json enough even in the DTH world?

I had a conversation with @davidfowl on jabbr indicating that the lock file might move all together in some dot folder which might mean the proposed resolved.json is already obsolete.

Is generating resolved.json even enough? How does it work with folks editing and saving project.json in VS?

[yishaigalatzer]

I'll let @davidfowl speak to the requirements of DNX, as he is much more familiar than me. But for project.json in UWP projects and PCLs, you have to write the project.lock.json completely as that is what msbuild picks up to include in the project.

The document in nuget.org describes just the none-DNX version, we are planning to merge the two in a future release, and will update the nuget documents when that happens.

To the flux question. Yes things are still changing.

[iancooper]

Here is the risk.

If you don't make it possible to swap the package manager out, you lock yourself into one package manager with the assumption that it will always provide what you need.

Given a 10 year lifecycle for this stuff, I would suggest that risk is High, with High impact if it happens.

The problem with hard-coded dependencies is that whilst they seem to make things simpler - why should anyone need to choose - they make our software inflexible to change.

This is essentially the Open-Closed principle at work. Open to extension and Closed to modification.

It's somewhat irrelevant if you need Paket, or if you believe you can make Paket obsolete.

Paket simply forms a useful test case for your ability to swap out the package manager in the future.

Given likely design risks from the hard dependency on one package manager I would suggest that enabling Paket has many architectural benefits that have nothing to do with Paket. Making the decision by comparing two package managers doesn't speak to the actual risk here.

[davidfowl]

I'm going to ignore the philosophical debates on this thread and stick to the technical side of things. Paket instructs the project system on which references to use via an msbuild import today, in the project.json world, that's a project.lock.json. We have an msbuild task for msbuild based projects and the dnx consume this file format which creates the appropriate environment for things like intellisense, compilation, copying the appropriate files for publish. It is the center of the universe (the lingua franca for references)

Seems like you're saying you want to be able to use paket for more than just msbuild based projects so you'll need to rethink what that means in this project.json only world. It means that:

• paket.dependencies/paket.references contains the package references
• project.json contains only project references
• project.lock.json file is what package generates from paket.references/dependencies

It's the moral equivalent to how to program msbuild to figure out the right references.

[forki]

@davidfowl thanks. That's the answer I was hoping to get. Last questions: in things like VS would it be enough to have the project.lock.json and a project.json without dependencies? Or is VS looking for that block? and do we need to replace the whole MSBuild import or is there a smaller part that handles dependencies and where we can hook into?

[yishaigalatzer]

You don't need to touch the msbuild script. Just create the lock file

Sent from my Windows Phone

---

From: Steffen Forkmannmailto:[email protected]
Sent: ‎11/‎9/‎2015 11:38 PM
To: fsprojects/Paketmailto:[email protected]
Cc: Yishai Galatzermailto:[email protected]
Subject: Re: [Paket] Paket support for vnext project files (#736)

@davidfowlhttps://github.com/davidfowl thanks. That's the answer I was hoping to get. Last questions: in things like VS would it be enough to have the project.lock.json and a project.json without dependencies? Or is VS looking for that block? and do we need to replace the whole MSBuild import or is there a smaller part that handles dependencies and where we can hook into?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/736#issuecomment-155345902.

[forki]

@yishaigalatzer but I assume there are events in the IDE that trigger an overwrite of the project.lock.json. Maybe something like adiing a file? Or is it only generated when someone touches the dependencies?

[davidfowl]

That's the answer I was hoping to get. Last questions: in things like VS would it be enough to have the project.lock.json and a project.json without dependencies?

Untested. There are a few things that would make the experience a bit rocky today. Automated package restore on save is one of them. You could imagine making that pluggable so we can run specific tools when restore is kicked off. We also put projects in the lock file today and we need a way to work that out as well.

Or is VS looking for that block? and do we need to replace the whole MSBuild import or is there a smaller part that handles dependencies and where we can hook into?

Like @yishaigalatzer said, the lock file is all you need.

[Mpdreamz]

Would something like:

 "dependencies": "external"

and e.g this in global.json

{
    "projects": [ "src" ],
    "restore" : "/.paket/paket.exe $file",
}

Be a quick win here? Existing tooling (vs/omnisharp) can keep calling dnu restore and the intend that an external tool is managing the deps is clearly communicated.

[nosami]

I haven't tested the latest bits, but at least with beta4, removing the dependencies out of project.json causes the following error on application start up.

10:59 $ ./scripts/Omnisharp
System.InvalidOperationException: Failed to resolve the following dependencies for target framework 'DNX,Version=v4.5.1':
   ICSharpCode.NRefactory 6.0.0-alpha3
   Microsoft.CodeAnalysis 1.0.0-rc2
   Microsoft.CodeAnalysis.CSharp 1.0.0-rc2
   Microsoft.CodeAnalysis.CSharp.Features 1.0.0-rc2
   Microsoft.CodeAnalysis.CSharp.Workspaces 1.0.0-rc2
   Microsoft.Composition 1.0.30
   Microsoft.Framework.Caching.Memory 1.0.0-beta4
   Microsoft.Framework.ConfigurationModel 1.0.0-beta4
   Microsoft.Framework.FileSystemGlobbing 1.0.0-beta4
   Microsoft.Framework.Logging.Console 1.0.0-beta4
   Microsoft.Framework.Runtime.Interfaces 1.0.0-beta4
   Newtonsoft.Json 7.0.1
   http://ScriptCs.Hosting  0.14.1

Searched Locations:
  /Users/jason/src/omnisharp-roslyn/src/{name}/project.json
  /Library/Frameworks/Mono.framework/Versions/4.2.1/lib/mono/4.5/{name}.dll
  /Library/Frameworks/Mono.framework/Versions/4.2.1/lib/mono/4.5/Facades/{name}.dll

Try running 'dnu restore'.

I don't know if it's feasible, but it would be nice if DNX could just use project.lock.json on start up for resolving dependencies. That way any tool that can write that file could be used with something like @Mpdreamz's syntax

[davidfowl]

@nosami beta4 is old. We're at RC1 now 😄 and the behavior is very different.

@Mpdreamz Sure the idea is fine but I'm not a fan of like that syntax. We'll look at something that is more natural. I'm not going to hack something in just to make paket work. It needs to gel with the rest of the system.

My suggestion would be to try it out and file issues for individual thing as you encounter them. The system is optimized for having dependencies all in a single file, not split across files. Also project references need to be taken care of which is another can of worms 😄.

[forki]

just tried to upgrade an new/empty beta8 project to rc2. Generated project.json.lock is 19705 lines.
I don't think that's scope of Paket.

[forki]

I mean I must be clearly doing something wrong here. 19k lines in a generated file. that cannot be correct, right? @Mpdreamz if you really want to lock versions (in paket's sense of a lock file) then you would have to commit that file, right? I mean project.json doesn't contain version numbers for transitive dependencies. What am I missing?

[Mpdreamz]

@forki thats what aspnet/dnx#2332 is about, to split the two responsibilities that project.lock.json currently has (caching of compilation info for DNX and resolved deps) in to two separate files.

[forki]

yes I was wondering if this is still the case.

[ctaggart]

Here is my understanding so far. MSBuild based projects have moved the dependencies out of the .proj files to a project.lock.json. The project.lock.json should not be committed to the repository and should be in .gitignore. A dnu or nuget restore uses a project.json and creates a project.lock.json. paket should support creating a project.lock.json instead of modifying the dependencies in the .proj files to support this new MSBuild world. Let me know where I am wrong. :)

[forki]

@ctaggart yes in principle we'd like to do that, but the rest of the infrastructure needs to understand that the top-level dependencies are not in maintained in project.json. And project.lock.json must only contain package specific information and not "project system" information. Otherwise Paket needs to do things that are out of scope.

[forki]

closed as "won't fix" see #1230 (comment)