Package Common Sense Mode

[bartelink]

Prompted by an IM from @JohannesRudolph:

NuGet is down; @ploeh was right again.
Just yesterday moved away from package restore (now checking in lib/nupkgs and using a local feed sourced from lib/nupkgs). For anyone else who needs a more reliable feed, you can try [https://nuget.appharbor.com/api/v2](official apphb mirror)

I'd like to see alongside the Paket restore mode, a mode in which Paket in a consistent way:

• works off just .nupkg files (downloaded source files would be serialized as .nupkgs)[https://github.com/File dependencies from arbitrary URLs  #114#issuecomment-56251073]
• .gitignore is set up such that only packages/*.nupkg is saved (the presence of the DLLs etc. is guaranteed idempotently by paket install) (but not subdirs - so it can tell if the files are not loaded, see below)

This could detect that the packages are not expanded (due to .nupkg files only being in /packages, and the canary one in /packages/.paket/expanded.nupkg not being present).

This would offer a middle way, addressing

• a lot of what The Package Restore People are scared of (lots of heavy commits creating source file folders with DLLs churning about)
• the need for commonsense (perf and resilience dictate that a repo should be freestanding)

---

I think an important thing is that in this mode, loading in VS is bad news - VS will silently half-load references and various other things and/or display other inconsistent behavior where DLLs do not exist.

Thus if each project had a target and/or failing import / InitialTargets or similar which could do a:

<Error Text="This project uses Paket compacted mode (See http://). Please usepaket installto prepare the repository for use from Visual Studion" Condition=" EXISTS( '$(SolutionRoot)/packages/.paket/expanded.nupkg' ) />

(Maybe just an Import with the message above as a filename?)

[forki]

http://fsprojects.github.io/Paket/nuget-dependencies.html#Path-sources this would probably work

[bartelink]

@forki But I dont want to manage such a mirror.

I want the packages in the packages folder. I want to check them into source control. But I don't want the DLLs and/or expanded content checked in.

And Paket has all the info/control it needs to work from the packages folder without any such mucking about and indirection on my behalf.

[forki]

just do the following:

• create subfolder nugets in your solution root
• put all nupkgs into that folder and commit it
• Use source ./nugets

still add /packages to gitignore.

Done

[bartelink]

And depending on the maturity of a project and/or the contributors' styles, one might just rely on NuGet caching for a bit and then 'flick the switch' by saying paket init-compact-mode, which immediately makes n .nupkgs ready to git commit -a -m "Now works on the plains and the planes"

[bartelink]

@forki But

1. I had to copy stuff into /nugets.
2. How will paket update know to write through the 'cache' that /nugets represents?
3. How will junk get deleted out of /nugets ?
4. How will I manage switchable caching of fssnip, myget, 2x nuget sources and github snippets in a uniform way?
5. How will I know that VS won't do a bodged half-load of the solution because a random subset of the DLLs are expanded?
6. How will I support intermittent failures (e.,g. github rate limiting)

[JohannesRudolph]

@forki that's exactly what I'm doing right now, but it would be awesome if Paket could automatically maintain the nugets folder for me, i.e. automatically mirroring the used packages from a remote feed.

[forki]

[bartelink]

@forki :) See above for more edits BTW

[forki]

so what you want is a second cache level which contains only installed packages and is customizable?

[bartelink]

@forki No. Paket needs Pakets.

It needs them to install from.

When it updates, it needs to coordinate the contents of the folder

There is zero value in having 2 copies of the nupkgs and/or the potential for them to lose sync.

The NuGet cache/download pipeline will, over time (as you've already accomodated) offer different resiliency options.

The download schemes from github / fssnip can do their own stuff too.

Paket has the overview of all dependencies and I'm not trying to introduce any new thing requiring new work. If you re-read the main post and/or I fix whatever confused you, you'll see... :D

The proposal is, in summary:

1. a .gitignore convention
2. a small snippet that gets put into each .proj which detects a canary / marker which signifies "don't expect the DLLs"

Some of my conventions presuppose some of the architecture in #154 in terms of unifying all the other things BUT right now, this scheme is extremely implementable for NuGet packages without any drastic messing about in the core impl of Paket.

OK, no more editing this comment!

UPDATE: cut/paste of side conversation

so what you want is a second cache level which contains only installed packages and is customizable?

I think second level cache describes it well
But we're using it as the one and only source for pacakge restores/installs

It will have the effect of being a second level cache

But its a primary thing which will be doing double duty

caches keep copies and coordinate stuff

#154 keeps everything consistently in one place

#166 says "add a .gitignore rule and inject stuff in VS which can tell you to go away if you only have the `.nupkgs'"

[forki]

I don't see why package restore and your .gitignore convention won't work together.

[bartelink]

@forki As in NuGet Package Restore ? Ah, might just work! .. HEY WAIT - there's no packages.config so how would/does VS know to unzip the .nupkg files first?

(Was typing) Of course, if someone can think of a way for (the same way new versions of NuGet work) for VS to trigger a paket install before attempting to load, we'd be golden and wouldnt need to emit such a message.

OK, so I don't know my .gitignore inside out - what's the simplest way of expressing "Include only .nupkg files directly under packages/. And exclude all other files under packages."

I assume that can be expressed as a single one liner rule. Does it make sense for Paket to look at .gitignore or is that a step too far? Or is it a doc / FAQ issue?

@vasily-kirichenko @rneatherway any ideas ?

[forki]

1. we should check if package folder is empty (except for nupkg) then we need to unzip (have to check. Maybe we already do this)
2. I don't know of a way to trigger MSBuild targets on VS load. But I don't think we really need this.
3. I prefer to have a separate .MD for this since I think a lot of people will like it

[bartelink]

@forki 1: Concievably a .nupkg could jump in from a branch switch so important to guarantee consistency

2: Re MSBuild targets main thing is to have a consistent can clear message that is searchable and/or points to doc and tells them to run paket install

In my experience its absolutely critical for people to never end up with an inconsistent project load as people are not always awake and/or understand WTH is going on in proj files.

But it def doesnt need to be high-tech. Just predictable - breaking, not bending if you will :)

3: Separate MD is cool with me - its def a concept that needs to be laid out with examples that I'm clearly not in a position to give!

And now, about that #154 elephant in the room :P

[forki]

can anyone try to figure out the gitignore pattern? /cc @theimowski

[agross]

Probably:

!*.nupkg

Alex

Alexander Groß
Tiny phone, tiny mail

On Thu, Sep 25, 2014 at 2:04 PM, Steffen Forkmann
[email protected] wrote:

can anyone try to figure out the gitignore pattern? /cc @theimowski

Reply to this email directly or view it on GitHub:
#166 (comment)

[JohannesRudolph]

Sorry for the brevity of previous comments, me and @bartelink were just throwing ideas around. Maybe we didn't even mean the same thing.

@forki I'm not sure whether it's a good idea to checking things from the packages folder. I think you were close by saying a "second level cache" with the nugets folder, but I think a better description may be a "curated, local mirror" of the packages from another feed.

Then you put something like this in your NuGet.config next to your solution (.sln)

<?xml version="1.0" encoding="utf-8"?>
<configuration>

  <packageSources>
    <add key="local-feed" value="lib/nuget-feed" />
  </packageSources>

  <activePackageSource>
    <add key="local-feed" value="lib/nuget-feed" />
  </activePackageSource>

</configuration>

I think nuget restore should automatically pick it up, if not explicitly pass it via -ConfigFile option to the nuget restore command. So on restore, you are retrieving all packages from that curated, local mirror. It's not only faster than using the official nuget feed, but also a lot more reliable (and gives you more reproducible builds as well when indirect dependencies are involved, which is the whole idea of Paket if I got that right).

Using the curated, local mirror approach you can keep using package restore like normal. That's how I do it currently, but I have to manually copy the nupkgs to my feed mirror directory. That's what I want paket to automate.

Now, if paket does already unpack/install nupkgs, committing them to the packages directroy may be enough. The .gitignore would then need to exclude every 2nd level subdirectory and below in the packages folder. With mercurial and glob syntax (not a git fan here, so may need translation):

packages/*/*/**

That leaves the nupkg files and the odd readme/license. I did actually try that approach first, committing packages//.nupkg, but that didn't play well using nuget restore, which would clame it had done its job while not unpacking the nupks.

So, as I see it, there are two choices for paket:

1. help maintaining the curated, local mirror feed
2. unpack nupkgs before build (which would probably require an msbuild hook somehow....)

Personally, I'm leaning towards the former.

[forki]

I only want to exclude them from within packages

2014-09-25 14:19 GMT+02:00 Alexander Groß [email protected]:

Probably:

!*.nupkg

Alex

Alexander Groß
Tiny phone, tiny mail

On Thu, Sep 25, 2014 at 2:04 PM, Steffen Forkmann
[email protected] wrote:

can anyone try to figure out the gitignore pattern? /cc @theimowski

Reply to this email directly or view it on GitHub:
#166 (comment)

—
Reply to this email directly or view it on GitHub
#166 (comment).

[agross]

Put the gitignore in the packages directory. Or did I misunderstand you?

Alex

Alexander Groß
Tiny phone, tiny mail

On Thu, Sep 25, 2014 at 2:34 PM, Steffen Forkmann
[email protected] wrote:

I only want to exclude them from within packages
2014-09-25 14:19 GMT+02:00 Alexander Groß [email protected]:

Probably:

!*.nupkg

Alex

Alexander Groß
Tiny phone, tiny mail

On Thu, Sep 25, 2014 at 2:04 PM, Steffen Forkmann
[email protected] wrote:

can anyone try to figure out the gitignore pattern? /cc @theimowski

Reply to this email directly or view it on GitHub:
#166 (comment)

—
Reply to this email directly or view it on GitHub
#166 (comment).

---

Reply to this email directly or view it on GitHub:
#166 (comment)

[forki]

but I want a global gitignore :-)

[agross]

Not a good idea, IMHO. Global breaks when you move things around. Violates DRY ;-)

Does this work?

packages/
!packages/*.nupkg

Alex

Alexander Groß
Tiny phone, tiny mail

On Thu, Sep 25, 2014 at 2:37 PM, Steffen Forkmann
[email protected] wrote:

but I want a global gitignore :-)

Reply to this email directly or view it on GitHub:
#166 (comment)

[bartelink]

@JohannesRudolph Lots of things to clarify. #125 makes a proposal to keep sources in a central place in a way that alludes to what you describe.

But the most fundamental thing here is that paket convert-from-nuget - the main way Paket makes sense to use:

1. paket convert-from-nuget
   • removes all packages.config files and replaces them with paket.references which list just file names
   • centralizes the list in /packet.dependencies
2. paket install looks at the lock file and
   • a: if no lock, go to 3. first
   • b: does a massive parallel download+unzip of any packages/*.nupkgs it can't see that the packages.references need according to the versions locked in paket.lock (but if the files are there it just uses them as paket update is entrusted with responsibility of getting all files consistent
   • c: visits all the .csproj/fsproj/vbproj files to ensure all references are right
3. paket update conceptually
   • figures out the graph and applicable versions
   • writes the paket.lock file
   • blasts away the entire packages directory - it knows everything required
   • jumps to 2

The proposal is that

• 2b guarantees to expand the zips so 2c will be correct if the .nupkg files are already there but not the DLLs
• 2c inserts stuff into the proj files that says "if the DLLs are not there, bail out and tell people to run paket install. The ONLY reason for this is for "clone in VS people"

There is no reliance on NuGet Package Restore and/or what versions 1-3 do in 9 releases of VS since VS2010SP1.

The other thing is that Paket needs to own an autoritative set of creds and feed locations across all feeds inc fssnip, github, local files etc.

BUT this is only used in 2b. Nothing else cares about the feeds. Paket Restore just works of a set of Pakets.

In step 1 of #154, "set of Pakets" means

1. the .nupkg files in the (unversioned) subdirs under /packages/
2. and special cased installation/managment of stuff that isnt from NuGet

In step 2 of #154, "set of Pakets" means

1. the .nupkg files in the (unversioned) subdirs under /packages/
2. 2. _and special cased installation/managment of stuff that isnt from NuGet_and step 2b stashes all stuff, including source files as .nupkg files too*

[bartelink]

@JohannesRudolph Based on the prev comments, I'd respond to your post as:

Sorry for the brevity of previous comments, me and @bartelink were just throwing ideas around. Maybe we didn't even mean the same thing.

V likely :P

Then you put something like this in your NuGet.config next to your solution (.sln)
OW, OLDE STYLE ANGLY BWAKETS, WE HAVE PARSERS AND DSLs!

No mirror defs

I think nuget restore should automatically pick it up, if not explicitly pass it via -ConfigFile option to the nuget restore command.

Nobody will be using NuGet restore.

So on restore, you are retrieving all packages from that curated, local mirror.

2b put them in packages. If you committed to source control they are there already. If not, 2b in a paket install will get them again.

It's not only faster than using the official nuget feed, but also a lot more reliable (and gives you more reproducible builds as well when indirect dependencies are involved, which is the whole idea of Paket if I got that right).

Have you tried Paket? There is nothing in NuGet's restore path that compares to the paket install perf.

The builds are reproducable as you say coz 3 computed the graph and stashed it in the paket.lock file

Using the curated, local mirror approach you can keep using package restore like normal. That's how I do it currently, but I have to manually copy the nupkgs to my feed mirror directory. That's what I want paket to automate.

Why? Does it handle multiple sources? If I am using an authenticated feed do I need to stash junk on my build machine? Does that accomodate non-NuGet sources?

Now, if paket does already unpack/install nupkgs, committing them to the packages directroy may be enough. The .gitignore would then need to exclude every 2nd level subdirectory and below in the packages folder. With mercurial and glob syntax (not a git fan here, so may need translation):

You didnt put ticks around angle brackets, I guess?

That leaves the nupkg files and the odd readme/license. I did actually try that approach first, committing packages//.nupkg, but that didn't play well using nuget restore, which would clame it had done its job while not unpacking the nupks.

Any Package restore stuff is stripped by paket convert-from-nuget. The block 2c puts in should hopefully neutralize any madness it wants to do. I agree the right thing needs to happen and we don't want VS trying to help and the causing a mess/

So, as I see it, there are two choices for paket:

1. help maintaining the curated, local mirror feed

No mirrors. No caches. No cache invalidation. No curation. Persistence of outcome of a deterministic download process. With a priviso for the stuff to already be there due to commiting to source control.

2. unpack nupkgs before build (which would probably require an msbuild hook somehow....)

@forki Could write a book NP. (And I've typed a book worth in contravention of @shanselman 's calls).

MORE IMPORTANTLY: Paket does all the stuff to unzip, edit the projs etc. In a flash. @forki has implemented the entire process, hard as that may be to imagine (I have to say I had the same reaction in terms of not considering ny of this stuff as being remotely feasible, but the counterexample is right here in the repo). (To be clear: there are plently other contributors and it might have taken an extra while without them, but the core of thie impl of this aspect is from @forki)

Personally, I'm leaning towards the former.

Still ? Need to do a skype call if so :P

[bartelink]

@agross I was fearing that might be the answer :( Maybe a minion needs to get Just One More Little Feature into git if you are right :P

[forki]

ok I tried. I don't get gitignore to work. Even if I put it into subdir.

If anyone want's to try it...

[JohannesRudolph]

@forki try this (see my above answer but I had the markdown wrong so it didn't show up). Works with mercurial.

packages/*/*/**

[bartelink]

@forki When Paket unzips, it lands the .nuspecs and misc files such as [Content_Types].xml beside the .nupkg

It would seem to me that @JohannesRudolph 's scheme can work if the install aspect consumes the .nuspec from the .nupkg on the fly and not send it out to disk (it's redundant anyway, or is it important for performance / cross checking of versions in early phases of the process) ?

[bartelink]

see also (possible) discussion (later) today https://jabbr.net/#/rooms/fsharp