make restore fast by copying and comparing the lockfile. #2675

matthid · 2017-08-26T19:51:40Z

I think checking & validating this hash in our msbuild targets file should make building large paket based solutions a lot faster...

matthid · 2017-08-26T19:53:13Z

I'm really wondering that this in itself doesn't make it faster at all (3 instead of 5 secs).

From my first profiling attempt it looks like Argu is taking the most time :/

matthid · 2017-08-26T19:53:39Z

Also the magic mode bootstrapper is probably eating quite some time there

…here.

0x53A · 2017-08-26T20:39:16Z

src/Paket/Program.fs

    try
+    let args = Environment.GetCommandLineArgs()
+    if args.Length = 1 && args.[0] = "restore" || args.Length = 2 && args.[0] = "--from-bootstrapper" && args.[1] = "restore" then


match args with | [| "restore" |] | [| "--from-bootstrapper" ; "restore" |]

?

yes looks better but might be slower :P
The main problem is generating the sha512 hash in msbuild anyway ;) I don't think what I want is even possible...

Ok fixed :)

matthid · 2017-08-26T21:37:00Z

Oh my god this seems to work

matthid · 2017-08-26T21:37:47Z

matth@DESKTOP-8VDEORF MINGW64 /c/proj/Paket (restore_speed)
$ PaketExePath=/c/proj/Paket/bin/paket.exe dotnet restore src/Paket.Core.preview3/Paket.Core.fsproj
  Paket version 5.91.0
  Performance:
   - Disk IO: 419 milliseconds
   - Runtime: 3 seconds
  Restore completed in 33.38 ms for C:\proj\Paket\src\Paket.Core.preview3\Paket.Core.fsproj.

matth@DESKTOP-8VDEORF MINGW64 /c/proj/Paket (restore_speed)
$ PaketExePath=/c/proj/Paket/bin/paket.exe dotnet restore src/Paket.Core.preview3/Paket.Core.fsproj
  Restore completed in 35.97 ms for C:\proj\Paket\src\Paket.Core.preview3\Paket.Core.fsproj.

matthid · 2017-08-26T22:03:30Z

IMHO this is good enough for now. It will hopefully help large netcore solutions and Paket.VisualStudio

matthid · 2017-08-26T22:19:38Z

WIP again, because netcore integration now needs to generate all files for all relevant frameworks beforehand

matthid · 2017-08-27T11:17:26Z

@alfonsogarciacaro I'm trying to change the restore process here a bit. In the future paket will write "obj",projectFile.Name + "." + targetFramework + ".references" instead of "obj",projectFile.Name + ".references". I know that Fable uses this file.

We can still write the old file for compat. The question is what should we write into it? What is the target-framework fable uses? netstandard16?

…t files and changed references files.

matthid · 2017-08-27T13:14:26Z

I guess the build will be red because this is a bootstrapping problem :/

matthid · 2017-08-27T13:18:24Z

src/Paket.Core/PublicAPI.fs

+            referencesFile.Groups
+            |> Seq.collect (fun kv -> kv.Value.NugetPackages)
+            |> Seq.map (fun i -> i.Name)
+            |> Set.ofSeq


This might be a slight change in behavior. I just look into the references file and assume all things there are direct deps (which might not be true but I planned to add a warning when transitives are listed in the references file - should be issue open somewhere)

matthid · 2017-08-27T13:19:42Z

src/Paket.Core/PublicAPI.fs

-    static member Init(directory) =  Dependencies.Init(directory,false)
-
-    /// Initialize paket.dependencies file in the given directory
-    static member Init(directory,fromBootstrapper) =


While technically a breaking change in the public API - I don't think anyone used these weird overloads.

alfonsogarciacaro · 2017-08-27T13:20:32Z

@matthid Fable 1.1.x uses netstandard1.6, I already publish a beta of Fable 1.2 which will use netstandard2.0, but I can make so the final Fable 1.2 looks for "obj",projectFile.Name + "." + targetFramework + ".references". So, if we want to keep compatibility with Fable 1.1.x for a time, it should be fine to use netstandard1.6.

matthid · 2017-08-27T13:21:58Z

@alfonsogarciacaro Thanks will update. So I will also assume that if no netstandard16 is in the project file we no longer need that file.

matthid · 2017-08-27T13:52:51Z

TLDR:

This PR improves the speed of paket restore
- by saving "cache" files and quickly checking if they are dirty
- by bypassing Argu (see Question about performance of Argu Argu#90)
This PR skips calling paket restore completely if we can detect in MSBuild that we are still up2date. (by using the same caching files as above)

alfonsogarciacaro · 2017-08-27T14:31:31Z

Awesome work, @matthid! Could you please check if this is going in the right direction? fable-compiler/Fable@112fcd3#diff-e47dbd37b262f2f0394b5704369d4b31R109

matthid · 2017-08-27T15:07:33Z

@alfonsogarciacaro Yes that looks exactly like it would do. Currently I'm thinking about renaming it to end with .paket.references but I haven't decided anything yet. Will let you know if I do :)

matthid · 2017-08-27T15:24:21Z

Ok green - finally :)

matthid · 2017-08-27T17:49:08Z

@Krzysztof-Cieslak Is Ionide calling paket restore directly or only dotnet restore? If the former we might want to take a look to benefit from this as well. I already updated fsprojects/Paket.VisualStudio#140 to benefit as well.

matthid · 2017-08-28T08:56:54Z

Results in FAKE:

Before:

DotnetRestore                      00:05:35.9467550
DotnetPackage                      00:06:52.9450041

https://ci.appveyor.com/project/matthid/fake-6w516/build/1.0.106

After:

DotnetRestore                       0 <- no longer required
DotnetPackage                       00:04:32.9611943

https://ci.appveyor.com/project/matthid/fake-6w516/build/1.0.148

alfonsogarciacaro · 2017-08-28T08:59:54Z

Awesome! How is the suffix finally? .references or .paket.references? When will this be released?

matthid · 2017-08-28T09:35:31Z

@alfonsogarciacaro Yes I think it makes sense, will try adding it later today. And I think this can be released. But before that someone (@forki) needs to answer how the target files are updated. Because older target files should be compatible with newer paket.exe but definitely not the other way around (So we need to know if this can happen)

Once we know that we can make a "real" release with this.

forki · 2017-08-28T09:36:28Z

Paket install extracts the target files. Am 28.08.2017 11:35 schrieb "Matthias Dittrich" <[email protected]>:

…

@alfonsogarciacaro <https://github.com/alfonsogarciacaro> Yes I think it makes sense, will try adding it later today. And I think this can be released. But before that someone ***@***.*** <https://github.com/forki>) needs to answer how the target files are updated. Because older target files should be compatible with newer paket.exe but definitely not the other way around (So we need to know if this can happen) Once we know that we can make a "real" release with this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2675 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNP0lPPtk-RC6fkNs0hC86Pd-kWrZks5sconjgaJpZM4PDmfw> .

matthid · 2017-08-28T09:38:29Z

Perfect, thanks that should work. Will add another PR with the renaming later.

make restore fast by saving a hash of the lockfile.

47af54a

matthid added 2 commits August 26, 2017 22:20

Cleanup -> remove fromBootstrapper as it was never actually used anyw…

9ced1f6

…here.

skip argu on simple restore

827c282

0x53A reviewed Aug 26, 2017

View reviewed changes

matthid added 2 commits August 26, 2017 22:54

MSBUILD don't call paket if the hash is up to date.

faee6e1

Instead of the hash store the full file contents

466d69a

cleanup the new code.

bba30bf

matthid added a commit to fsprojects/Paket.VisualStudio that referenced this pull request Aug 26, 2017

only use 'restore all' to benefit from fsprojects/Paket#2675

593f09d

matthid changed the title ~~[WIP] make restore fast by saving a hash of the lockfile.~~ make restore fast by saving a hash of the lockfile. Aug 26, 2017

matthid changed the title ~~make restore fast by saving a hash of the lockfile.~~ make restore fast by copying and comparing the lockfile. Aug 26, 2017

matthid changed the title ~~make restore fast by copying and comparing the lockfile.~~ [WIP] make restore fast by copying and comparing the lockfile. Aug 26, 2017

do all the restore work globally and add detection for changed projec…

358af94

…t files and changed references files.

matthid changed the title ~~[WIP] make restore fast by copying and comparing the lockfile.~~ make restore fast by copying and comparing the lockfile. Aug 27, 2017

matthid commented Aug 27, 2017

View reviewed changes

matthid added 4 commits August 27, 2017 15:29

change compat code

3ad3050

fix build by immediatly using the newly build binary

4e5d55b

use full path

9da9428

cleanup, we don't actually use this for now..

f6d6ce4

matthid requested a review from forki August 27, 2017 13:49

matthid mentioned this pull request Aug 27, 2017

Add PaketReference MSBuild item in projects instead of paket.references #2606

Open

more bootstrapping

66d456c

matthid added 2 commits August 27, 2017 16:45

release notes

717081c

handle special case of empty references file.

3520c6c

we need to delete stale data.

cb4cd32

matthid merged commit cb4cd32 into master Aug 27, 2017

matthid mentioned this pull request Aug 28, 2017

dotnet pack/fix-nuspecs is still slow #2678

Open

matthid deleted the restore_speed branch July 2, 2019 17:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make restore fast by copying and comparing the lockfile. #2675

make restore fast by copying and comparing the lockfile. #2675

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

0x53A Aug 26, 2017

matthid Aug 26, 2017

matthid Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid Aug 27, 2017

matthid Aug 27, 2017

alfonsogarciacaro commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

alfonsogarciacaro commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 28, 2017

alfonsogarciacaro commented Aug 28, 2017

matthid commented Aug 28, 2017

forki commented Aug 28, 2017 via email

matthid commented Aug 28, 2017

make restore fast by copying and comparing the lockfile. #2675

make restore fast by copying and comparing the lockfile. #2675

Conversation

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

0x53A Aug 26, 2017

Choose a reason for hiding this comment

matthid Aug 26, 2017

Choose a reason for hiding this comment

matthid Aug 26, 2017

Choose a reason for hiding this comment

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 26, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid Aug 27, 2017

Choose a reason for hiding this comment

matthid Aug 27, 2017

Choose a reason for hiding this comment

alfonsogarciacaro commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

alfonsogarciacaro commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 27, 2017

matthid commented Aug 28, 2017

alfonsogarciacaro commented Aug 28, 2017

matthid commented Aug 28, 2017

forki commented Aug 28, 2017 via email

matthid commented Aug 28, 2017