[Ready] Make paket build on a clean windows #2664

0x53A · 2017-08-25T20:34:08Z

Windows

Install Windows 10 Pro from ISO
install git
clone paket
run build.cmd

(Note: IntegrationTests were skipped in this screenshot (because they take so much time), but they were also green!

Linux

We still rely on a global mono and a global MSBuild, because the bundled version is not compatible with linux and crashes hard on startup.

Notes:

msbuild is provided by 'RoslynTools.MSBuild'
ReferenceAssemblies are currently checked-in. I need to either find a nuget package, or create one myself. (I had planned to use 'RoslynTools.ReferenceAssemblies', but that doesn't contain the ones for 4.5) I created a nuget package using https://github.com/jaredpar/xcopy-msbuild. This contains only the two directories it needs (NETFramework\v4.5 to compile against net45 AND .NETPortable\v4.5 to consume portable assemblies, for example FsCheck)
MSBuild doesn't seem to support Directory.build.props in combination with old-sdk projects? I had wanted to use that to set FrameworkPathOverride, but it didn't work until I manually modified all fsprojs Only the first Directory.Build.props is used, and there was already one in /src/. Also, Directory.build.targets does not yet work: Directory.build.targets not imported when cross-targeting dotnet/msbuild#1721
~~there is something funky with referencing portable stuff, I had to disable the FsCheck tests, or I would get the following errors:~~ fixed after adding ref-assembly directory .NETPortable\v4.5

matthid

Is there another way than to add all those binaries to the repository?

0x53A · 2017-08-25T21:39:25Z

what binaries do you mean? ;D

matthid · 2017-08-25T21:41:59Z

paket.dependencies

@@ -44,3 +44,9 @@ group Build
  nuget ILRepack

  github fsharp/FAKE modules/Octokit/Octokit.fsx
+
+  nuget 0x53A.ReferenceAssemblies.net45 0.1.0


So this is for net45 support, correct? because in the future this is no longer required as soon as we are fine with net461 compat?

No, these are the reference assemblies. When we move to net461 we need to update to the correct reference assemblies for that.

The reference assemblies are the "contract assemblies" that the compiler compiles against. They are metadata only.

I used https://github.com/jaredpar/xcopy-msbuild, this basically just zipped up the content of C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5

But once we are on net461 all we need is the netstandard build, so we could drop that in theory?
I don't think this is the correct thing to do long term (though I haven't really decided yet).

I thought we will have all the reference assemblies as "official" nuget packages now?

I haven't found any reference assemblies as nuget packages.

Otherwise I'm pretty sure the roslyn build would use these, instead of packaging this stuff up themselves.

Can you elaborate/link the RoslynTools.ReferenceAssemblies you are referring to above (I cannot find it)?

source https://dotnet.myget.org/F/roslyn-tools/api/v3/index.json

And also why it doesn't work for us (just trying to figure out if we can make it work or if it is viable for other scenarios)?

It does not contain the ref-assemblies for 4.5:

Hm, maybe I can use this?

Will try it ...

Maybe should just upgrade to net46? I'm not sure about the portable stuff could be a subset..

So, I did a test, and I am pretty sure we need .NETFramework\v4.5 to compile against net45 AND .NETPortable\v4.5 to consume portable assemblies, for example FsCheck.

I have created a package that contains only these two. Reason: it is smaller.

0x53A · 2017-08-26T01:04:40Z

The tool dotnet-mergenupkg was compiled against netcore1.0, so we need the netcore1.0.x runtime to execute it ...

The issue is that Fake only allows the installation of a single runtime, I tried installing two after each other but the second deleted the first.

0x53A · 2017-08-26T01:12:26Z

Hi @enricosada, do you think it would be possible to update dotnet-mergenupkg and provide a netcoreapp2.0 binary?

Otherwise usage in the Sdk 2.x is a bit clumsy, because I still need to install the 1.0.x runtime in parallel somehow.

I've hacked together a netcoreapp2 version, so I can continue testing this: enricosada/dotnet-mergenupkg@master...0x53A:master

0x53A · 2017-08-26T01:49:50Z

matthid

Way better :). I like the direction of this - It probably will make contributing easier.

matthid · 2017-08-26T08:40:15Z

paket.dependencies

@@ -44,3 +44,10 @@ group Build
  nuget ILRepack

  github fsharp/FAKE modules/Octokit/Octokit.fsx
+
+  nuget 0x53A.ReferenceAssemblies.Paket 0.2.0


So every project now bundles their own reference assemblies :P

Or put differently, if you think this is a good way forward (which I think it is). We might want to have another documented project on how this stuff works (Then others can use it and read about it). Anyway that's just a suggestion (I'd like to maybe add that to FAKE as well)

Yeah, I would prefer to use an official package, also.

The one advantage this has is that it only bundles the directories it actually needs, so the size is a bit smaller. I could have given it a specific name like "ReferenceAssemblies.Net45", but this way it is clear that it is used by paket and will change as soon as we update.

Building the package is dead easy, clone https://github.com/jaredpar/xcopy-msbuild, modify which frameworks to include https://github.com/jaredpar/xcopy-msbuild/blob/006e1bb65d031854f38eaac251b5f799ce281e37/build-reference-assemblies.ps1#L23-L30
and run the powershell script.

Maybe we should add that as comment then?

matthid · 2017-08-26T09:00:43Z

build.cmd

@@ -5,4 +5,7 @@ if errorlevel 1 (
  exit /b %errorlevel%
 )

+set MSBUILD=packages\build\RoslynTools.MSBuild\tools\msbuild


Maybe we could set that in the FAKE script.

sure, that may also fix unix, which currently fails (because it still runs the bundled msbuild).

How would you set it inside Fake? Just set the env vars? or is there some config switch?

Good question. I think in FAKE 4 the way would be to change the mutable field with the path.
But yeah I don't know how a good FAKE API for this would look like...

0x53A · 2017-08-26T15:00:54Z

Well, i guess I need to set up a linux vm to fix this :\

0x53A · 2017-08-26T16:02:50Z

what a bunch of crap:

matthid · 2017-08-26T16:12:46Z

Oh yeah ;)

0x53A · 2017-08-26T16:15:09Z

well, one issue was that ubuntu 17.x still references some ancient 4.x mono ...

0x53A · 2017-08-26T16:15:32Z

does paket not use caching on unix?

Maybe I'm wrong ,but it seems to download everything every time

0x53A · 2017-08-26T16:16:01Z

wait - paket now finished successfully???????

I only had to start the build script 20 times!

forki · 2017-08-26T16:17:02Z

Mono? It's short for "Mondays..." Am 26.08.2017 6:12 nachm. schrieb "Matthias Dittrich" < [email protected]>:

…

Oh yeah ;) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2664 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNF7tVkZB690l4jhPKSqCxaYsd36tks5scEP-gaJpZM4PDIgM> .

matthid · 2017-08-26T16:17:31Z

Well, if we don't find a global cache we fallback to some local directory. Maybe that is what you are seeing?

0x53A · 2017-08-26T16:21:27Z

Idk, I was probably confused. Each time I started it, it downloaded some, then failed. But each time it probably got a little bit further...

You may want to mute this thread, i will probably rage a bit until I either have this fixed, or given up ;)

forki · 2017-08-26T16:23:17Z

Some of the integration tests also run -f Am 26.08.2017 6:21 nachm. schrieb "Lukas Rieger" <[email protected]>:

…

Idk, I was probably confused. Each time I started it, it downloaded some, then failed. But each time it probably got a little bit further... You may want to mute this thread, i will probably rage a bit until I either have this fixed, or given up ;) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2664 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNMcoe6ZZ52njOZ6airOfqufNwo38ks5scEYIgaJpZM4PDIgM> .

0x53A · 2017-08-26T16:24:21Z

good news: I got it to use the msbuild from the package.

bad news: it crashes hard on startup

0x53A · 2017-08-26T16:51:30Z

Ready from my side, as soon as AppVeyor and at least one travis leg are green

matthid

I'd love to see a bit less "WIP"-Dependencies but overall that's a good step forward.

matthid · 2017-08-26T17:11:46Z

paket.dependencies

@@ -44,3 +44,10 @@ group Build
  nuget ILRepack

  github fsharp/FAKE modules/Octokit/Octokit.fsx
+
+  nuget 0x53A.ReferenceAssemblies.Paket 0.2.0


Maybe we should add that as comment then?

matthid · 2017-08-26T17:12:26Z

src/Directory.Build.props

@@ -1,9 +1,12 @@
 <Project>
+  <!-- MSBuild only includes the first Directory.Build.props, so we need to manually include the root one -->
+  <Import Project="$(MSBuildThisFileDirectory)..\Directory.Build.props" />


Maybe / like above?

msbuild does a normalization, the relevant change was the casing. I think I will go back to backslash for everything

Maybe we should add that as comment then?

I had a comment in the actual commit, but yes, will add a source comment

0x53A · 2017-08-26T17:22:18Z

I'd love to see a bit less "WIP"-Dependencies but overall that's a good step forward.

The version numbers are arbitrary. If it makes you feel better I can push a 1.0 right now ;D

If you want, we can also change to a "non-branded" package that is owned by you guys or the fsfoundation.

"0x53A.ReferenceAssemblies.Paket" was just so that I could get started.

Similar, I forked dotnet-mergenupkg to dotnet-0x53A-mergenupkg so that I could work on this now, we can wait with actually pulling this until enrico has updated his original package to netcoreapp2.0.

matthid · 2017-08-26T17:25:01Z

Note that it's not a blocker for me :). I trust you to cleanup eventually. That's why approved the changes ;)

0x53A · 2017-08-26T17:31:30Z

There wasn't any real need for this PR, I mostly wanted to test this out so I can also move my work solution to this model. So it is up to you whether we pull it now and clean up later, or wait a bit.

forki · 2017-08-26T17:42:08Z

The version numbers are arbitrary

Welcome to netcore ecosystem ;-)

matthid · 2017-08-26T18:02:29Z

build.fsx

@@ -364,7 +364,7 @@ Target "MergeDotnetCoreIntoNuget" (fun _ ->

    let runTool = runCmdIn "tools" dotnetExePath

-    runTool """mergenupkg --source "%s" --other "%s" --framework netstandard1.6 """ nupkg netcoreNupkg
+    runTool """0x53A-mergenupkg --source "%s" --other "%s" --framework netstandard1.6 """ nupkg netcoreNupkg


I think this fixes #2645. This is bad. We didn't even realize that we still depend on SDK1.0 to be installed. Thanks :)

You are welcome ;)

0x53A · 2017-08-26T18:15:50Z

* acquire msbuild via RoslynTools.MSBuild * acquire reference assemblies via a self-built package based on https://github.com/jaredpar/xcopy-msbuild * use a fork of dotnet-mergenupkg which was recompiled against netcoreapp2.0 (original was netcoreapp1.0) to remove dependency on 1.x runtime

0x53A · 2017-08-27T22:48:27Z

Note: I would like to add an actual CI leg for this, but afaik AppVeyor does not provide "clean" windows machines, so I don't know of a way to test this.

matthid · 2017-08-27T22:50:05Z

Maybe with travis a minimalistic netcore only build?

0x53A · 2017-08-27T22:51:03Z

Maybe with travis a minimalistic netcore only build?

On linux, I still rely on the system-installed msbuild because the one from RoslynTools.MSBuild crashes hard

matthid · 2017-08-27T22:56:42Z

"dotnet msbuild" could work. I mean there is a bit of work to do to make this happen but it's not too unrealistic as paket/fake are basically ported. I think the only thing I know is that the netcore version of paket cannot push for some reason

0x53A · 2017-08-27T22:57:47Z

Officially, the dotnet cli does NOT support building old-sdk projects.

I would wait until we have ported paket completely to new-sdk.

matthid · 2017-08-27T23:00:03Z

then we need to update sooner or later anyway, because I really want to get rid of mono. We can still have tests and stuff working with mono but the core stuff will probably move soon :)

0x53A · 2017-08-27T23:07:28Z

yes, it would be great to move paket itself to netcoreapp, maybe even standalone so the dotnet runtime does not need to be installed.

Do you know what the status of the linker is? Single-file standalone would be awesome.

matthid · 2017-08-28T05:41:35Z

I would be so happy already if stuff would be working, so I didn't think about a linker yet ;)

But yeah it would be awesome.

enricosada · 2017-09-01T12:48:55Z

@0x53A released dotnet-mergenupkg v2.0.0

This support both sdk 1.0 and 2.0 as cli tool

forki · 2017-09-03T08:19:00Z

AWESOME!!!!

0x53A force-pushed the bootstrapping branch from 60b1dae to 32dc45e Compare August 25, 2017 21:16

matthid requested changes Aug 25, 2017

View reviewed changes

0x53A force-pushed the bootstrapping branch from 171c045 to 8ebcd19 Compare August 25, 2017 21:39

matthid reviewed Aug 25, 2017

View reviewed changes

0x53A closed this Aug 26, 2017

0x53A reopened this Aug 26, 2017

0x53A mentioned this pull request Aug 26, 2017

I need a strategy to install multiple dotnet versions in parallel fsprojects/FAKE#1640

Closed

0x53A mentioned this pull request Aug 26, 2017

please provide a netcoreapp2.0 binary enricosada/dotnet-mergenupkg#3

Closed

matthid requested changes Aug 26, 2017

View reviewed changes

0x53A force-pushed the bootstrapping branch from d5caf1a to 5ffc150 Compare August 26, 2017 14:42

matthid reviewed Aug 26, 2017

View reviewed changes

matthid approved these changes Aug 26, 2017

View reviewed changes

matthid reviewed Aug 26, 2017

View reviewed changes

0x53A closed this Aug 26, 2017

0x53A reopened this Aug 26, 2017

0x53A added 2 commits August 26, 2017 20:33

run paket update

6d343b3

0x53A force-pushed the bootstrapping branch from b71f50f to 385a821 Compare August 26, 2017 18:33

Merge branch 'master' into bootstrapping

f79116a

0x53A changed the title ~~[WIP] Make paket build on a clean windows~~ [Ready] Make paket build on a clean windows Aug 27, 2017

update README: add notes about prerequisites

99d915c

Merge remote-tracking branch 'upstream/master' into bootstrapping

2d8be87

forki merged commit 2d8be87 into fsprojects:master Sep 3, 2017

0x53A deleted the bootstrapping branch September 3, 2017 15:16

0x53A mentioned this pull request Sep 17, 2017

[Build] dotnet-0x53A-mergenupkg => dotnet-mergenupkg #2773

Merged

0x53A mentioned this pull request May 3, 2019

enable support for .NET Fx 4.6.1 <-> .NET Standard 2 #3447

Open

[Ready] Make paket build on a clean windows #2664

[Ready] Make paket build on a clean windows #2664

Conversation

0x53A commented Aug 25, 2017 • edited Loading

Windows

Linux

matthid left a comment

Choose a reason for hiding this comment

0x53A commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x53A commented Aug 26, 2017

0x53A commented Aug 26, 2017 • edited Loading

0x53A commented Aug 26, 2017

matthid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x53A commented Aug 26, 2017

0x53A commented Aug 26, 2017

matthid commented Aug 26, 2017

0x53A commented Aug 26, 2017

0x53A commented Aug 26, 2017

0x53A commented Aug 26, 2017

forki commented Aug 26, 2017 via email

matthid commented Aug 26, 2017

0x53A commented Aug 26, 2017

forki commented Aug 26, 2017 via email

0x53A commented Aug 26, 2017

0x53A commented Aug 26, 2017

matthid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x53A commented Aug 26, 2017

matthid commented Aug 26, 2017

0x53A commented Aug 26, 2017

forki commented Aug 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x53A commented Aug 26, 2017

0x53A commented Aug 27, 2017

matthid commented Aug 27, 2017

0x53A commented Aug 27, 2017

matthid commented Aug 27, 2017

0x53A commented Aug 27, 2017 • edited Loading

matthid commented Aug 27, 2017

0x53A commented Aug 27, 2017

matthid commented Aug 28, 2017

enricosada commented Sep 1, 2017

forki commented Sep 3, 2017

0x53A commented Aug 25, 2017 •

edited

Loading

0x53A commented Aug 26, 2017 •

edited

Loading

0x53A commented Aug 27, 2017 •

edited

Loading