Lockfile ordering to surface direct vs indirect references

[bartelink]

This proposal is a rework of #123 which takes into consideration the proposal for #154. To be honest the crickets over there are making me do this, with the aim of illustrating the value of unifying the concepts:

Say I have a deps file which has (as paket simplify would derive from a F# MVC 5 wizard- created WebApi 2 [only, no MVC element] app) :

nuget bootstrap
nuget hyprlinkr
nuget Microsoft.AspNet.WebApi

And my tests use:

nuget unquote
nuget xunit 1.9.2

Current paket.lock format:

This results in a lock file like this today:

NUGET
  remote: http://nuget.org/api/v2
  specs:
    Microsoft.AspNet.WebApi (5.2.2)
      Microsoft.AspNet.WebApi.WebHost (>= 5.2.2, <  5.3.0)
    Microsoft.AspNet.WebApi.Client (5.2.2)
      Newtonsoft.Json (>= 6.0.4)
      Newtonsoft.Json (>= 6.0.4)
      Microsoft.Net.Http (>= 2.2.22)
    Microsoft.AspNet.WebApi.Core (5.2.2)
      Microsoft.AspNet.WebApi.Client (>= 5.2.2)
    Microsoft.AspNet.WebApi.WebHost (5.2.2)
      Microsoft.AspNet.WebApi.Core (>= 5.2.2, <  5.3.0)
    Microsoft.Bcl (1.1.9)
      Microsoft.Bcl.Build (>= 1.0.14)
    Microsoft.Bcl.Build (1.0.21)
    Microsoft.Net.Http (2.2.28)
      Microsoft.Bcl (>= 1.1.9)
      Microsoft.Bcl.Build (>= 1.0.14)
    Newtonsoft.Json (6.0.5)
    Unquote (2.2.2)
    bootstrap (3.2.0)
      jquery (>= 1.9.0)
    hyprlinkr (1.1.1)
      Microsoft.AspNet.WebApi.Core (>= 4.0.20710.0)
    jquery (2.1.1)
    xunit (1.9.2)

Proposed Output

If we have a tokenizer (#129) that understands the #154 syntax, and the resolver generates similar Pakets, then one can [conceptually] 'easily' produce:

sources
  nuget  nugetv2  http://nuget.org/api/v2

nuget bootstrap 3.2.0
  dependency jquery (>= 1.9.0)
nuget hyprlinkr 1.1.1
  dependency Microsoft.AspNet.WebApi.Core (>= 4.0.20710.0)
nuget Microsoft.AspNet.WebApi 5.2.2
  dependency Microsoft.AspNet.WebApi.WebHost (>= 5.2.2, <  5.3.0)
nuget Unquote 2.2.2
nuget xunit 1.9.2

nuget jquery 2.1.1
  // Used by bootstrap (>= 1.9.0)   
nuget Microsoft.AspNet.WebApi.Core 5.2.2
  dependency Microsoft.AspNet.WebApi.Client (>= 5.2.2)
  dependency Microsoft.Net.Http (>= 2.2.22)
  // Used by hyprlinkr (>= 4.0.20710.0)
  // Used by Microsoft.AspNet.WebApi.WebHost (>= 5.2.2, <  5.3.0)
nuget Microsoft.AspNet.WebApi.WebHost 5.2.2
  dependency Microsoft.AspNet.WebApi.Core (>= 5.2.2, <  5.3.0)
  // Used by Microsoft.AspNet.WebApi (>= 5.2.2, <  5.3.0)

nuget Microsoft.AspNet.WebApi.Client 5.2.2
  dependency Newtonsoft.Json (>= 6.0.4)
  dependency Microsoft.Net.Http (>= 2.2.22)
  // Used by Microsoft.AspNet.WebApi.Core (>= 5.2.2)
nuget Microsoft.Net.Http 2.2.28
  dependency Microsoft.Bcl (>= 1.1.9)
  dependency Microsoft.Bcl.Build (>= 1.0.14)
  // Used by Microsoft.AspNet.WebApi.Core (>= 2.2.22)

nuget Microsoft.Bcl 1.1.9
  // Used by Microsoft.Net.Http (>= 1.1.9)
nuget Microsoft.Bcl.Build 1.0.21
  // Used by Microsoft.Bcl (>= 1.0.14)
  // Used by Microsoft.Net.Http (>= 1.0.14)
nuget Newtonsoft.Json 6.0.5
  // Used by Microsoft.AspNet.WebApi.Client (>= 6.0.4)

Algorithm

1. Top level entries in the dep file ('primaries') as paket simplify would produce. Sort by <PaketId>and emit with a blank line.
2. Follow dependency entries one level down. Sort by <PaketId> and emit with a blank line. (jquery, Microsoft.AspNet.WebApi.Core, 'Microsoft.AspNet.WebApi.WebHost` )
3. Repeat 2. until no dependency items in the working set (Microsoft.AspNet.WebApi.Client, Microsoft.Net.Http)
4. Remainder goes in a separate group (e.g. Newtonsoft.Json, Microsoft.Bcl.Build, Microsoft.Bcl). Sort by <PaketId> and emit with a blank line.

Production rule used to compact output

nuget <NuGetPackageName> (<Ver>) in the above is a rendering pattern match / substitution that replaces paket specs with their primary source dependency id (in the case of a nugetv2 source, a <NuGetPackageId>) matching the paket's`:

paket bootstrap 
  nuget bootstrap 3.2.0

=>

nuget bootstrap 3.2.0

Comments please!

[forki]

You make it look like the dependencies are a tree or forest. Actually we have a graph that may contain cycles.

[bartelink]

Is that a bad thing? What does it imply/are you trying to imply? What do you think makes sense instead?

I fully appreciate they are a graph. The algorithm is clear about what it does. The prior stuff in #123 makes a lot of this clear. Recall we started with mentions of hierarchy but I gave examples where it didnt make sense. Have a look at the examples in #123 before you consider this useless, not scalable or similar ;)

So.... what's your point? I believe this achieves the following well:

• show most important stuff at the top
• show least important stuff at the bottom
• shows successive stages of derivation
• no redundant information
• aligns with structure in Pakets #154
• aligns well with simplify

It shouldnt matter but:

• not a crazy jump from the original
• relatively easy to impl
• remains easy to parse

I can't say I cant imagine anything better but I think its a good textual representation of a complex graph (which even visually would be hard to pull something out of).

[forki]

No I didn't say it's useless.
On Sep 27, 2014 11:33 PM, "bartelink" [email protected] wrote:

Is that a bad thing? What does it imply/are you trying to imply? What do
you think makes sense instead?

I fully appreciate they are a graph. The algorithm is clear about what it
does. The prior stuff in #123
#123 makes a lot of this
clear. Recall we started with mentions of hierarchy but I gave examples
where it didnt make sense. Have a look at the examples in #123
#123 before you consider this
useless, not scalable or similar ;)

So.... what's your point? I believe this achieves the following well:

• show most important stuff at the top
• show least important stuff at the bottom
• shows successive stages of derivation
• no redundant information
• aligns with structure in Pakets #154
  Pakets #154
• aligns well with simplify

It shouldnt matter but:

• not a crazy jump from the original easy to impl
• remains easy to parse

I can't say I cant imagine anything better but I think its a good
textual representation of a complex graph (which even visually would be
hard to pull something out of).

—
Reply to this email directly or view it on GitHub
#165 (comment).

[bartelink]

I know. I'm trying to spark debate and/or trigger you gathering your thoughts as to what way we might achieve a better outcome in either a completely different way or by evolving this.

I'd like to see an interesting question like "how will it deal with this set" or "in this case, the rendering hides the core fact one is after when analysing the set" ? I just know that the first time I tried to answer questions with the lockfile, Ctrl-F3 was extremely frustrating and it took me ages to manually derive the hierarchy (in fairness doing it by clicking about in the NuGet search windows in VS would be grossly worse and would have been my default option - same way I never formally realised how much time I've spent pressing update in VS). I think today I'd use Nuget Package Explorer btu every time I've wanted to answer a question recently, I've played it thorough this algorithm and it seems (with my confirmation bias anyway!) to be reasonably solid.

I have played 3 or 4 things through this. If you look athe the MVC example, it actually produces quite a different sequence to WebApi. This tells me there are some cases where it's going to produce wierd results. Actually, there's prob a tree which will look like

A
  dependency B

B
  dependency C

C
  dependency D

D
  dependency E

E

which might trigger insight which either says the algorithm is not going to work or trigger an insight we can use to improve and not waste impl time (obv one could easily derive a rule for this case but it would be far better to have a real world tree from a complex case - perhaps I need to try http://github.com/AutoFixture/AutoFixture - it's got quite a lot of stuff and some subtle pinning rules that I happen to be familiar with).

[bartelink]

Re remote dependencies these can be shown in here, esp with #154 refinements - remote dependenceis can even be sufaced as a separate paket-entry.

V neat concept.