NuGet pack on projects with paket dependencies

[mavnn]

If you NuGet pack a proj file with an associated nuspec file (same filename in same directory), NuGet automatically populates a lot of the nuspec fields from the proj file.

Most of these work fine, regardless of whether the proj file has NuGet or Paket dependencies - but the automatic dependency resolution obviously doesn't!

I'm looking at writing something that updates the nuspec dependency nodes; the logic NuGet uses is documented here: http://docs.nuget.org/docs/reference/command-line-reference#Pack_Command and is in general quite sane.

All of this to say: would you like this to be (an optional) part of the Paket code, or shall I write it as something independent?

And any pointers on whether we do anything around project references at the moment?

[forki]

1. https://github.com/tpetricek/FSharp.Formatting/blob/master/build.fsx#L153 works with Nuget and Paket
2. Paket.Core allows you to query the lock file

[mavnn]

1. is only really convenient for solutions with a single packaged output. We regularly have solutions that generate tens of dlls which need packaging, and I don't think it's that unusual. Also - those dependencies had to be added manually, but could have been calculated automatically from Paket data and project references. NuGet does this already, and we leverage it heavily in larger solutions.
2. It does indeed, I've now discovered! Which leads to...

A proposal. @vasily-kirichenko , I know you're using paket in larger solutions as well, I'd love your input.

Packaging Paket based solutions

For single dll projects, project scaffold gives a good example of how a nuget package can be created. However, it does not scale well for solutions with many output NuGet packages.

To avoid confusion for people moving from NuGet, and frankly also because it seems a pretty sensible idea, I'd recommend a Paket based option that closely follows the NuGet "From a Project" flow.

This relies on a nuspec file in the same directory as a *proj file; when you pack, you pack based on the project file which then loads up the nuspec, and substitutes in token values based on the build output of the project. It will also optionally create a dependencies node in the nuspec with required nuget dependencies.

To achieve the same functionality in a Paket based solution, we will need the following steps (steps in brackets are only required when generating the dependencies node):

• Check if the project has an associated nuspec file
  • skip if not
  • check which tokens have been used - we might be able to skip some of the other steps
• Scan the project file to determine:
  • The output file
  • (Project references where the target project has a nuspec file)
• Scan the output file (with reflection) to load its assembly info
  • Obtain assembly version - or assembly informational version for pre-release versions
  • Obtain the values for all other tokens used in the nuspec file
• (Scan the paket.dependencies and paket.lock files to determine minimum versions of nuget dependencies)
• (Recursively scan project reference project files to work out:)
  • (Versions of output files via reflection if they have a nuspec file of their own)
• Create a new nupkg file based on the nuspec but...
• with the project output file added as a lib
• (any project references without a nuspec file copied to the same lib folder)
• a dependency node injected with all paket dependencies and all project references where that project has a nuspec file
• all tokens replaced with metadata from the build output

It would also be nice to have some solution level functionality (as ripple used to); this would allow commands like "package this solution" and would avoid duplicate work in solutions that make heavy use of project references.

This is a big old bunch of work, most of it only peripherally related to Paket's main purpose. Because of that I'm tempted to suggest that I start work on it as a separate project ("Packen"?) that takes a dependency on Paket.Core; I'm more than happy to keep it part of Paket if people prefer though.

[forki]

/cc @agross @Vidarls

[Vidarls]

@mavnn Check out fsprojects/ProjectScaffold#101 (and fsprojects/ProjectScaffold#116 for current version of the code) for my take on how to do this through the project scaffold project. (It also enables you to have interproject dependencies as nuget dependencies for projects that uses a core + modules architecture (like NancyFx or Simple.Data) where you maintain a set of slim nuget packages rather than "fat" ones.

As to the question of where this kind of code should live: The closer you are to the build process, the more opinonated you can be, and the code will be simpler to create. IE in project scaffold you can expect a specific naming convention, you know where you expect to find binaries and packages. When you remove yourself from that very opinionated space, to say Paket.Core you have to cater for everybody elses opinions on all sorts of stuff, and the job will be much more difficult.

My goal with the (admittedly still unmerged) pull request to Project Scaffold is to get required helper functions ( @forki implemented quite a few during the process) into dependent libraries like Paket and possibly Fake, and have an opinonated fully functional implementation as part of Project Scaffold. All the helpers should make it easy for anyone to adapt to their own needs if they do not agree with the opinions of Project Scaffold.

That said, I might be having to low ambitions, and getting some Super-Duper-Happy-Path(tm) way of packaging up a project in Paket might be well worth the effort.

[mavnn]

@Vidarls I think this functionality should exist independently of a particular project type. Whilst I like (and use) project scaffold for some of our open source projects we have our own project conventions for in house code that don't play nicely with the scaffold way. Whether it should be provided by Paket or by a separate project that depends on the Paket libraries is a much more open question in my mind. Either way, I'm sure the code already written in your pull request will simplify the creation!

[kYann]

Hi,

I'm also interested in a "paket pack *.proj" solution because we already have our own build process and solution architecture, and this option would allow a drop in replacement of nuget.

[mavnn]

@forki unless anyone objects, I'm going to start putting together a pack command as part of paket (as it will require heavy use of Paket.Core regardless). I'll try and spec out a format for a nuspec replacement (with an auto-convert feature as part of the nuget convert, preferably) and go from there.

[forki]

yes. pack command inside paket would be really nice

[kYann]

Great !

[mavnn]

Okay... keeping some design notes in here so people can see.

nupkg files are zip files based on OPC: https://en.wikipedia.org/wiki/Open_Packaging_Conventions

Apart from the OPC conventions, they also store their content in convention based directories as specified here: http://docs.nuget.org/docs/creating-packages/creating-and-publishing-a-package

As well as both of the above, they contain a certain amount of metadata. This includes at root level a valid nuspec file using the http://docs.nuget.org/docs/creating-packages/creating-and-publishing-a-package schema. Note that the nuspec files consumed by NuGet.exe do not normally use this namespace. This nuspec file has a defined relationship following OPC in the /_rels/.rels file. The nupkg also includes a "package" directory with some additional metadata in a format I haven't yet determined.

All of this metadata would (it appears) be generated by the Packaging implementations in System.IO.Packaging (.net 3.0+) - except that for some reason these classes are in WindowsBase.dll, making them unsuitable for use in Paket currently (is that correct, @forki ?).

So:

1. Determine what the package directory is and what needs to go in it
2. Prove I can write valid trial nupkg files that both nuget and paket will consume with DotNetZip
3. Add a nice dsl for for paket.package files
4. Add dependency inference for the package from a combination of project files and paket environment
5. Write a nuspec -> paket.package converter
6. Add to paket code base
7. Profit

That's a bit more work than I was hoping for, I must admit. But I think it's needed for getting multi-dll output projects completely NuGet free. Otherwise updating the output dependencies becomes untenable.

[forki]

Yep pack command will be a lot of work. That's why we didn't started this yet.

[mavnn]

Well... we have working OPC zip files that nuget reads correctly

[mavnn]

Experimentation taking place here: https://github.com/mavnn/NupkgCreator

[kYann]

✨ 👍 ✨

[mavnn]

So; we've now reached the stage where we can turn things like:

type file
id My.Thing
authors Bob McBob
description
    A longer description
    on two lines.
version
    1.0
dependencies
     FSharp.Core 4.3.1
     My.OtherThing

Into a valid nupkg (see here), although you have to provide actual content files by hand in code at the moment.

Next up: a syntax for specifying contents. Then, generating metadata from a project if type project is specified at the top of the file.

[mavnn]

Oh, and the parsing is insanely inefficient at the moment, but I'll get things working before I bother sorting that...

[mavnn]

@forki what would you like the extension of packaging config files to be? ".pack"? ".packen"?

I'm worried ".pack" might conflict with other things; it sounds a little too generic.

[forki]

probably something like paket.template

[mavnn]

Hmm. That works for the files that are next to a project file. But if you want to handle the packaging manually rather than using project inference, you probably want to put multiple files in the same directory.

For example

├───src
│   └───My.Proj
│       ├───My.Proj.fsproj
│       └───paket.template // works
└───templates
    ├───My.ManualNuget.template // not so good with fixed file name
    └───My.ManualNuget2.template

I can see it would be nice to keep the paket in file name convention though.

paket.[optional.id.]template?

[forki]

yes it's the same with references files. If you don't have MyProject.paket.references we use paket.references

[mavnn]

Excellent! Hadn't realized there was a convention already. MyName.paket.template it is.

[forki]

see http://fsprojects.github.io/Paket/references-files.html#File-name-conventions

[mavnn]

Brief consultation moment: I'm planning on assuming that people will want to package after a "Release" build for project based packaging - any objections?

On default VS projects it makes a difference on where to look for the output directory.

[mavnn]

For reference, the other alternative would be to use the default configuration in the proj file, but this is nearly always Debug and I think that would break the principle of least surprise.

[agross]

In my projects Debug and Release put their output in the same non-standard directory. Just FYI.

Alex

Alexander Groß
Tiny phone, tiny mail

On Tue, Feb 10, 2015 at 5:23 PM, Michael Newton [email protected]
wrote:

For reference, the other alternative would be to use the default configuration in the proj file, but this is nearly always Debug and I think that would break the principle of least surprise.

Reply to this email directly or view it on GitHub:
#521 (comment)

[mavnn]

Yes - that's how project scaffold works as well. I'm trying to infer which
OutputPath node to use, not assuming the standard /bin/Release.

On 10 February 2015 at 16:35, Alexander Groß [email protected]
wrote:

In my projects Debug and Release put their output in the same non-standard
directory. Just FYI.

Alex

Alexander Groß
Tiny phone, tiny mail

On Tue, Feb 10, 2015 at 5:23 PM, Michael Newton [email protected]
wrote:

For reference, the other alternative would be to use the default
configuration in the proj file, but this is nearly always Debug and I think

that would break the principle of least surprise.

Reply to this email directly or view it on GitHub:
#521 (comment)

—
Reply to this email directly or view it on GitHub
#521 (comment).

[mavnn]

@forki given you're made the convert from nuget aspect of this a separate issue in #621 I think this can be closed now.