Impossible to target Xamarin.Mac .NET 4.5 Framework

[mteper]

Situation

Xamarin.Mac projects can be set up to target one of the following Frameworks:

Xamarin.Mac Mobile Framework, defined as:

The Xamarin.Mac Framework is based on the same tuned .NET framework used by Xamarin.iOS and Xamarin.Android. This is a recommended framework. It provides smaller average binaries due to better linking behavior.

Xamarin.Mac .NET 4.5 Framework, which is defined as:

The Xamarin.Mac Packaged .NET 4.5 is a subset of the desktop .NET framework that Xamarin considers the highest quality and supportable long term. This allows you to consume standard desktop assemblies but produces larger application bundles. This is another recommended framework.

The list of supported assemblies is here: http://developer.xamarin.com/guides/mac/advanced_topics/assemblies/

Unsupported Framework, e.g. Mono / .NET 4.5, but the UI has this to say:

Building Xamarin.Mac applications using the system mono desktop .NET framework is useful in porting legacy applications but is not officially supported by Xamarin. Migrating to another Xamarin.Mac Framework is strongly recommended.

Problem

Only one of the three targets above (mobile) maps to the xamarinmac TFM. When a project set up to target either of the other two framework targets adds or updates a NuGet package that contains binaries for "xamarinmac", it instead picks up the "net45" ones. This is problematic for libraries like ReactiveUI, Splat, Xamarin.Insights, etc. The onus then falls on the developer to manually update path references in their .csproj files, which causes undue friction and errors, especially on larger teams.

Here's the corresponding Xamarin issue: https://bugzilla.xamarin.com/show_bug.cgi?id=21338

Now that Xamarin is part of Microsoft, can we please get this pain addressed?

[rrelyea]

can you please find the right Xamarin contact here?

[mrward]

@chamons should be the Xamarin contact for this.

[ghuntley]

There are three distinct problems which need resolving:

• XM45 does not have a TFM.
• A Xamarin Mac 45 application when consuming a library will consume NET45 ignoring XamarinMac. This feels wrong, if a package has a XamarinMac (aka Mobile) assembly then it should it not be chosen over the NET45 assembly?
• What should the Xamarin Mac 45 application resolution order be when there is a NET45, XamarinMac and XM45 assembly in a package?

Repro over at https://github.com/ghuntley/repros/tree/master/nuget/Issue2662 and https://github.com/ghuntley/repros/blob/master/nuget/Issue2662/Issue2662.nuspec

[chamons]

Hello everyone. I'm the Xamarin.Mac lead from Xamarin. 👋

Sorry I didn't get the summons before.

The breakdown from @mteper is pretty solid from what I can tell. We have things documented here:

https://developer.xamarin.com/guides/mac/advanced_topics/target-framework/

My detailed knowledge of nuget is lacking, so I apologies in advance for my ignorance.

A bit of a history lesson here will help explain how we got into the situation we are in:

When we created the Unified, we originally (in XM 1.12) only had the "Mobile" target framework for preview. However, a significant number of people were consuming a a lot of nuget packages, which knew nothing about "xamarinmac". This was a world of pain, and unacceptable.

XM 4.5 was our last minute "solution". It lies, at least a bit, about what it is, so that we can consume many of those packages, which actually just work under the hood.

However, if you "know" about XM 4.5 and want to provide special builds, say because your "desktop" build depends on Windows specific version, there is no good way to sub them in.

Is there a way (my end or your end?) where we can say something like:

if there is a xamarinmac45 build use that, if not fall back to the "normal desktop" build we're already using?

[harikmenon]

@joelverhagen. We talked about this a bit internally. Adding Joel, He can take a look at this issue and help identify what exactly is needed to fix this issue.

[joelverhagen]

@ghuntley, could you give me a little more background on your repro? I have not worked with Xamarin Studio very much so it's not clear to me how to reproduce the issue. Also -- another silly question -- what is the "XM45" TFM?

[mteper]

@joelverhagen when you open the solution in Xamarin Studio, right-click on the project and select Options. Then, switch to "General" tab under "Build". You will see the framework target options I referred to in the initial case description.

XM45 refers to Xamarin.Mac .NET 4.5 Framework target.

[ghuntley]

@joelverhagen Xamarin internally calls "Xamarin.Mac .NET 4.5 Framework" by the codename XM45, a proper identifier/moniker has not been assigned by MSFT or Xamarin. This is the internal codename, potentially could/should be renamed?

Inside that repro you will find two dummy applications:

Mac45App             = Xamarin XM45 Application 
MacMobileApp         = Xamarin Mac 

Additionally there are three dummy libraries.

MacMobileLibrary     = xamarinios TFM
Mac45Library (XM45)  = unassigned TFM
Net45Library         = net45 TFM

The end goal is to make XM45 recognised by NuGet and design how/if/when a XM45 library is installed depending on the scenarios above into the two applications.

[joelverhagen]

Got it. It seems to me that you would like to add a new xm TFM with new compatibilities with existing TFMs (e.g. net45).

Adding a TFM to NuGet is relatively straightforward. The list of frameworks that NuGet knows about is in this file, so this is where the work should be started:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Frameworks/DefaultFrameworkMappings.cs

This file also contains all of the framework equivalencies, compatibilities, subsets, etc.

FrameworkReducer is the guy that takes the TFMs made available by a package and chooses the most appropriate one based on a specified project framework. The unit tests for FrameworkReducer illustrate these TFM precedences you are asking about.

Precedence is an angle that must be very well unit tested to avoid regressions as new TFMs are added.

[joelverhagen]

I have knowledge in this area so I could work on the implementation, probably within the next couple weeks. You are also free to work on a PR which NuGet team can review.

I understand these Xamarin TFMs as abstract terms but I am pretty ignorant about when or why they are used. Let's nail down some details.

Clarifying questions for points already made:

1. Can we agree on a different TFM for the Xamarin.Mac .NET 4.5 framework? xm is rather cryptic. I am thinking xamarinmacnet plus the version number. For example, xm45 would change to xamarinmacnet45. The long name can be Xamarin.Mac.NET. Thoughts?
2. What relationship does this new framework have with net and xamarinmac in terms of compatibility? What I can glean from above is:
   1. net45 should not support xamarinmacnet or xamarinmac.
   2. xamarinmac does not support net45 or xamarinmacnet45.
   3. xamarinmacnet45 should support net45 and xamarinmac. Additionally, you want xamarinmac to be preferred over net45.

New questions:

1. For the existing xamarinmac TFM, what are the possible versions and what do they mean? I have seen xamarinmac10 and xamarinmac20 in the wild. How do these framework versions relate to the compatibilities above?
2. What framework versions do we have to consider for xamarinmacnet? Is xamarinmacnet45 the only one we are considering?
3. So far we have make all Xamarin frameworks support the latest netstandard (1.6 right now) as well as the latest deprecated dotnet (5.6, which is equivalent to netstandard1.5). Is this okay?

Note that NuGet has a generic rule that all frameworks support themselves when the package framework version is less than or equal to the project framework version. For example, net45 supports net40.

[chamons]

We are looking at renaming the "XM 4.5" target framework already. I agree that it is a terrible name. I'm ok with the internal name not matching that. :)

I am honestly unsure on all of the existing nuget identifiers. I think how it works is:

• xamarinmac10 = Xamarin.Mac Classic - This is XamMac.dll and standard net45 (we pull from whatever is installed on system). We're not making any real changes to Classic and suggest most people migrate to Unified, so I'm ok making no changes here.
• xamarinmac20 = Xamarin.Mac Mobile? - Xamarin.Mac.dll and our mobile libs at (Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/Xamarin.Mac/).

I believe your assertions (i, ii, iii) are correct. net45 is it's own thing. xamarinmac/xamarinmac20 should be it's own thing (it is limited like iOS). xamarinmacnet45 (or whatever we call it) should fall back to net45 .

On your questions:

1. I believe xamarinmac10 and xamarinmac20 should not change. They are not part of this "lying" use case.
2. Right now xamarinmacnet45 is the only case for fallback that we know of (thought it isn't tied to 4.5 which is why the name is bad).
3. I believe that is fine.

[joelverhagen]

I believe your assertions (i, ii, iii) are correct. net45 is it's own thing. xamarinmac/xamarinmac20 should be it's own thing (it is limited like iOS). xamarinmacnet45 (or whatever we call it) should fall back to net45

So xamarinmacnet45 should not support any version xamarinmac? I am not sure what you mean by xamarinmac being it's own thing.

My assertion "iii" above was:

xamarinmacnet45 should support net45 and xamarinmac. Additionally, you want xamarinmac to be preferred over net45.

I want to make sure these details are exactly right 😄

[chamons]

I'll spell it out in non-nuget first, since I'm 99% sure on that, then do the translation:

Xamarin.Mac Mobile (/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/Xamarin.Mac/) - Really only is safe pulling it builds just built for it. It is a rather limited subset of the BCL. If you pull in net45 here, then you often get brokenness.

Xamarin.Mac XM 4.5 (Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/4.5/) - This is the guy that pretends to be a "normal desktop" assembly. This is where we'd love the "if there is a special build for it, use that, else fall back to standard net45" behavior.

I believe mobile = xamarinmac and xm45 = xamarinmacnet45 and thus:

xamarinmac just supports itself and does not fall back (this is the current behavior, right?)
xamarinmacnet45 supports itself and falls back to net45 (this is the new behavior change, right?)

Does that sound reasonable? Clear as mud?

[mteper]

@chamons,

What about XM45 falling back to xamarinmac first, before net45? For example, Xamarin Insights ships with a xamarinmac dll but doesn't (yet?) ship an XM45 one. We would want to fall back to xamarinmac over net45 in this scenario, right?

[joelverhagen]

xamarinmac just supports itself and does not fall back (this is the current behavior, right?)

Correct. xamarinmac is a known TFM in NuGet and frameworks always support themselves (if the version is less than or equal).

xamarinmacnet45 supports itself and falls back to net45 (this is the new behavior change, right?)

Yes, NuGet does not know about the xamarinmacnet moniker at all (or xm for that matter). It seems like we are arriving at a consensus 👍

What about XM45 falling back to xamarinmac first, before net45?

@mteper and @chamons, this is your call as you know the details of these compatibilities at runtime. If xamarinmac has an API surface area that is a strict subset of the xamarinmacnet or net45, then consider this compatibility. I'll let you be the final judge since there may be other framework differences that I don't know about.

[chamons]

Hmm. That is a hard one @mteper Mobile is almost, but not a complete subset of XM 4.5. That is not a scenario that we've considered / tested / know even works.

However, it would be rather useful, as the insights use was shows.

Let me think and get back to this thread later today...

[mteper]

@chamons wrote,

That is not a scenario that we've considered / tested / know even works.

FWIW, we have it working in production for ReactiveUI, Splat, and Xamarin.Insights (all of which target xamarinmac). Perhaps it would be helpful to enumerate those things in XM Mobile that lie outside XM45?

[chamons]

So in terms of just number of assemblies, I believe XM 4.5 is an actual superset of Mobile:

https://github.com/xamarin/xamarin-macios/blob/master/builds/Makefile#L102

However, we build the mono class library very differently:

https://github.com/mono/mono/blob/master/mcs/build/profiles/xammac.make
https://github.com/mono/mono/blob/master/mcs/build/profiles/xammac_net_4_5.make

where Mobile is built, well like the "mobile" iOS profile and XM 45 is built like desktop, except for places where that is unavoidable.

I rather like the idea of XM 4.5 falling back on Mobile, as it really makes the life of nuget devs easy if they want to support XM and they can fit inside mobile. Let me poke a few people here and see if there is some disaster I'm not considering.

[joelverhagen]

To give you an idea of the sort of changes I plan on making for this new TFM, take a look at my feature branch here:
https://github.com/NuGet/NuGet.Client/compare/jver-xamarintfm?expand=1

I will not put this as a PR until I've heard back from you @chamons about xamarinmacnet45 supporting xamarinmac. Oh, and one detail is that I made the compatibility such that any version of xamarinmacnet will support xamarinmac up to version 2, e.g. xamarinmac, xamarinmac2, but not xamarinmac3.

[chamons]

I can't think of a reason why this isn't a great idea, and my colleagues at work (who also don't know nuget super well) can't come up with a counterexample either.

👍

When would this change reach a nuget near me? I'd like to tweet / post on our forums to let Xamarin.Mac users know about the change.

[joelverhagen]

How do you normally consume NuGet? Do you use the .NET CLI experience? NuGet.exe?

[chamons]

More specifically, if not all of net45APIs are available in xamarinmacnet, won't some net45 packages just blow up?

Yep. That's the problem with lying. Sometimes you get caught.

However, there are a huge number of nuget packages out there that don't know about XM at all. If we don't lie, we can't use any of them.

The XM45 target framework was\is an imperfect solution to an unhappy problem.

I'm happy to hear better solutions, doubly so if they are backwards compatible. :)

[chamons]

If net45 is really the closest matching API surface area, then netstandard1.1 should be supported, (not netstandard1.6, as with other Xamarin TFMs) due to the fact that net45 itself supports only netstandard1.1. Any objections?

I do not have enough context with nuget to understand the question enough to answer it safely.

What XM45 ships in almost all cases is what mono builds in their "normal" build. For example for System.dll in XM45 we just use: "System.dll.sources" (https://github.com/mono/mono/blob/master/mcs/class/System/xammac_net_4_5_System.dll.sources)

I have no idea what that maps to in nuget land.

[joelverhagen]

In the project.json world, this problem is addressed using imports. Suppose your consuming project is targeting xamarinmacnet and wants to pull in a net45 package. In NuGet we could choose to have no "baked in" compatibility between the two, but if developers wanted to take the risk and pull in a net45 package they could target xamarinmacnet with a net45 "imports" directive. This would tell NuGet to fallback to net45 if a package does not provide a xamarinmacnet assembly.

I am not familiar with the project system you are working with. Could project.json be used?

[joelverhagen]

(edited to clarify my wording 😓)

[chamons]

All Xamarin projects (that I know of) are the standard msbuild project format processed by xbuild.

[clairernovotny]

AFAIK, project.json isn't yet supported by Xamarin Studio...in the stable channel. I've heard that it is available in the alpha channel.

You should be able to use project.json with any project type though for nuget references, I blogged about how.

[mhutch]

@chamons the term "project.json" is overloaded. It can be used with MSBuild, in which case it's just a replacement for packages.config, not the build system. As Oren mentioned, we support this in the alpha channel.

I don't see how this can work in the form proposed - the 4.5 frameworks for XM don't have a TFM of their own for NuGet to match on, and we can't add one without breaking the ability to reference 4.5 library projects. Solving this is going to be difficult.

[zone117x]

Is this still being pursued? This has been a painful problem.

[joelverhagen]

/cc @rrelyea, could someone on the client side take ownership of this? I've been speaking with @chamons and @mrward offline so I can hand off any and all information I have to whoever picks this up on client side.

[xavier-rigau]

I am having a similar issue. When I have a Xamarin.Mac .NET 4.5 app install my nuget package it uses the net45 targetFramework or the uap10.0 if listed in the nuspec. If I don't remove the uap10.0 group my nuget package won't install at all.

When I don’t comment the UWP section the error I get is this: Package 'Microsoft.NETCore.Jit.1.0.3' does not exist in folder '/Users/xavierr/svn/XamarinNew/packages'

However the log shows it was retrieved.

https://bugzilla.xamarin.com/show_bug.cgi?id=21338#c9

[zone117x]

This has continued to be a really painful issue for us. Surprised it has not been fixed after all this time. Anyway to get this prioritized?
Trying to use aspnetcore for a self hosted http server and the packages cannot be installed on either of the "supported" Xamarin.Mac frameworks. Forcing us to use unsupported net451 which comes with this own set of issues..

[jainaashish]

I dont see this going any where. There was a open PR for over an year but then no further movement so we had to close that PR. I'm not sure what NuGet can do here except this scenario is supported for new PackageReference world with net core project with multiple target frameworks or with fallback target frameworks.

I'm not sure we're going to do much about packages.config scenarios to support this but if Xamarin wants to take that ownership and suggest what to do there then we can coordinate.

[mattleibow]

Reopened this with NuGet/NuGet.Client#2572

[fschwiet]

+1

This is going to be a problem for anyone trying to use Xamarin.Mac from a console app per https://docs.microsoft.com/en-us/xamarin/mac/app-fundamentals/console.