Should there be a .Net Standard 2.0 version of Reflection.Emit?

[svick]

System.Reflection.Emit is not part of .Net Standard, but there are packages that let you use it from a .Net Standard library (specifically, System.Reflection.Emit and System.Reflection.Emit.Lightweight). But those packages do not have a .Net Standard 2.0 version, only .Net Standard 1.x versions.

This has some implications:

• .Net Standard 2.0 libraries can't use typeBuilder.CreateType() (even though this code works both on .Net Framework 4.6.1 and .Net Core 2.0) and have to use typeBuilder.CreateTypeInfo().AsType() instead. It's possible there are also other APIs like this.
• .Net Standard 2.0 libraries that want to use Reflection.Emit still have to use .Net Standard 1.x-style System.* packages.

These issues would be solved if a .Net Standard 2.0 version was added to Reflection.Emit packages. Is that something that would be worth doing?

Though both of these are fairly small quibbles, so I'm not sure how much value would doing this add.

[karelz]

@ericstj @weshaggard do you know if it is on purpose this way, or an oversight?

[ericstj]

It was intentional. For libraries like this that have jagged support and no portable implementation we stopped shipping packages. The expectation is that you target a specific framework if you need it. For some we have brought back packages with land mines (throwing implementations on some platforms). Folks could propose we add it back but this one in particular has had some stong debate in the past due to lack of support on all AOT based .net frameworks. /cc @terrajobst

[svick]

@ericstj

The expectation is that you target a specific framework if you need it.

How am I supposed to learn about that? As far as I can tell, Reflection.Emit is not reported by platform-compat. And when I google "reflection emit .Net Standard", the only result on the first page talking about whether you should use it that way is a tweet from July 2017 by @terrajobst suggesting that you should do it (maybe the situation changed since then?).

[ericstj]

I’ll let @terrajobst respond to that, perhaps he was thinking about the nuget gaurdrails feature we had in 1.x and the 1.x packages. With those we had jagged implementations and build time support for messaging that things aren’t supported. I’m not particularly leaning one way or the other here just explaining the way things are working at the moment. I think if we can get sign off from folks on adding back a package with a platform not supported implementation we could do so.

FWIW anything that’s available in a framework but isn’t in netstandard or in a library/package with a netstandard asset means you need to target the framework to get it. That’s all I meant with my expectation comment.

[svick]

@ericstj

anything that’s available in a framework but isn’t in netstandard or in a library/package with a netstandard asset means you need to target the framework to get it. That’s all I meant with my expectation comment.

But here the package (with .Net Standard 1.x assets) does exist. I don't think you can just stop updating a package and assume people won't use it anymore. At the very least, you should have documentation explaining the situation.

[weshaggard]

The biggest trouble with System.Reflection.Emit is that we don't have a way to provide the library in a netstandard way because of TypeInfo (see https://github.com/dotnet/corefx/issues/14334). The existing package is targeting netstandard1.1 but it is a lie and will only work on .NET Framework and .NET Core because of the tight coupling we have. We hacked it to work by playing InternalsVisibleTo tricks to give it access to TypeInfo's constructors, but that same trick is what makes it not work on a general .NET Standard platform. We decided to not propagate the error further by hacking netstandard2.0 in the same way and instead stopped producing the package and made it platform specific.

Thanks for raising the issue @svick and we should use this issue to document the trouble with the package.

[akoeplinger]

What about unlisting the no longer updated packages on NuGet so it's clear they are not recommended?

[weshaggard]

That is a reasonable suggestion. I just unlisted all the versions of https://www.nuget.org/packages/System.Reflection.Emit/ and https://www.nuget.org/packages/System.Reflection.Emit.Lightweight

[masonwheeler]

This is very needed. Not being able to build and save assemblies is currently my dotnet/corefx#1 biggest blocker for updating to .NET Core, and I'd really like to see getting this very fundamental API up to speed made a higher priority than working on new features.

[weshaggard]

@masonwheeler just to call out you can use Reflection.Emit if you are writing .NET Core app or library you just cannot use it while writing .NET Standard library currently.

[masonwheeler]

@weshaggard are you sure? I just checked the repo, and it appears that not only is AssemblyBuilder.Save not implemented yet, it doesn't even exist at all on Core! (I might be wrong; I'm on my phone and GitHub's mobile interface isn't the best, but that's what it looks like.)

[weshaggard]

@masonwheeler you're correct that we don't support AssemblyBuilder.Save in .NET Core that is being tracked by another issue https://github.com/dotnet/corefx/issues/4491.

[masonwheeler]

There it is! I thought I had seen that issue before, but searching only turned up this one. :P

But yeah. Without the ability to actually emit your results, you can't really say that you can use Reflection.Emit on Core. ☹️

[SergeyKhutornoy]

@weshaggard before the System.Reflection.Emit.Lightweight package being unlisted on nuget.org it was possible to get Selenium.WebDriver package in PowerShell 5 (for .Net Framework) and PowerShell Core 6 (for .Net Core 2.0) on Windows 10 using the following command: Install-Package Selenium.WebDriver -Destination $PSScriptRoot -Force -ForceBootstrap

Now this command cannot download all the Selenium package dependencies for both PowerShell versions. It fails because System.Reflection.Emit.Lightweight.4.3.0 package is unlisted on nuget.org.
Error: Install-Package : Unable to find dependent package(s) (System.Reflection.Emit.Lightweight)

Could you advise on how to fix this issue?

[weshaggard]

@SergeyKhutornoy are you calling "Install-Package" in VS from the Package Manager Console? If so that correctly installs the package for me. If you are calling Install-Package from Powershell that is a different kind of package management. I actually just tried that locally and I get a different error (Install-Package : No match was found for the specified search criteria and package name 'Selenium.WebDriver'.) I'm not familiar with that package management system so I'm not sure how to work around it. One thing you could try is to explicitly install System.Reflection.Emit.Lightweight 4.3.0 first and then see if it works. If that doesn't work is there a reason you cannot use VS or nuget tools to install the package?

[bording]

Unlisting this package is a mistake, and not providing a .NET Standard 2.0 version is a mistake.

We're about to ship a major new version of our product, NServiceBus, and we're targeting netstandard2.0. We also have a dependency on System.Reflection.Emit and System.Reflection.Emit.Lightweight. I had originally intended to target .NET Framework and .NET Core separately, but a twitter conversation with @terrajobst combined with discovering these packages were available led me to change plans and target netstandard2.0 instead.

[atykhyy]

I don't mind not being able to save dynamic assemblies so much - I can always test the logic in a .NET Framework TFM - but DynamicMethod is incredibly useful and very widely used to generate reflection thunks, converter delegates etc. System.Reflection.Emit.* packages have upwards of 20,000,000 downloads.

[eiriktsarpalis]

Please bring back a way to reference DynamicMethod in netstandard2.0 libraries.

[danielcrenna]

How do we use DynamicMethod in .NET Core now that it is unlisted as a package?

[svick]

@danielcrenna You don't need the package to use DynamicMethod on .Net Core, since it's built-in. You only needed the package to use it on .Net Standard.

[FransBouma]

I could build my runtime (which is released already) on .netstandard 2.0 using System.Reflection.Emit.Lightweight 4.3 as I use DynamicMethod. I now see the build process pulls the package from offline cache and not from nuget. If the offline cache is hosed, I can't build the code again for .NETstandard. (Hosing the cache does make the build process pull the package again, as the package is still there, albeit unlisted, so it's not a showstopper technically. It is semantically though).

Looking here: https://apisof.net/catalog/System.Reflection.Emit.DynamicMethod what's the conclusion I have to take? That it's in NS1.6? Or in some in-between dimension where 'platform extensions' are valid?

So iow: As I target .netstandard2.0 and use DynamicMethod, I can make a reference to System.Reflection.Emit.Lightweight although it's unlisted, and supported in ns1.6 (?), but this feels really off: it feels as a liability as I now depend on a package that's unlisted (and I then have to hope unlisted packages are kept there till the end of time. ) Sad thing is: there's no alternative other than depending on an unlisted package that is only possible because one knows the exact name.

// @terrajobst

[FransBouma]

To add: EF Core 2.1 has a dependency on Castle.Core (https://www.nuget.org/packages/Castle.Core/) for its proxies, which depends on System.Reflection.Emit.

Isn't it wise for all involved to have this package (and Emit.Lightweight) to simply be enlisted again?

[cincuranet]

@FransBouma strictly speaking EF Core Proxies has the dependency, not EF Core itself. But it's a mess anyway.

In fact FirebirdClient has dependency on System.Reflection.Emit. And what to do now, right?

[seesharper]

While it is understandable that full AOT platforms can't allow new types to be generated at runtime, it is more of a surprise that we can't support System.Reflection.Emit.Lightweight on these platforms.
LambdaExpression.Compile()works across all platforms even if it is interpreted on AOT.

I am the author of the LightInject DI library and that is using Reflection.Emitand DynamicMethod to generate code at runtime. This was all good until these other platforms started to emerge. SilverLight, WinRT, iOS and so on. So what to do? What I did was to "shim" the DynamicMethod so that OpCodes are translated into expressions. There is a kind of a 1-1 relationship between Opcodes and expressions so that is not at all that hard.

Take a look here for the implementation.
https://github.com/seesharper/LightInject/blob/a01be40607761d9b446dc4acad37d7f717742975/src/LightInject/LightInject.cs#L4483

Note that I don't implement all the Opcodes. Only the ones I need.

My point is that it should be possible to do this for all OpCodes and then enable A LOT of libraries to still target netstandard2.0. Most libraries that uses Reflection.Emit does not generate new types. They simply generate code through the DynamicMethod

Also note that DynamicProxy, Moq and other libraries that generates types at runtime cannot target netstandard2.0. They must be netcoreapp since they basically rely on AssemblyBuilder with friends
So bottom line is that the System.Reflection.Emit.Lightweight package can never be removed from NuGet entirely as netstandard2.0. That would be the LeftPad story all over again

My two cents

[jbogard]

Is there a list somewhere of the TFMs that do support System.Reflection.Emit? I've just added net45 and netcoreapp2.0 explicit TFMs to my project, but I'm sure I'm missing some.

[seesharper]

@jbogard All full framework TFM's should be good in addition to the netcoreapp TFM's. Is this in AutoMapper?

[mythz]

This seems like a poorly thought out and communicated decision with far reaching consequences which appears to have been made with very little involvement from the community it impacts, surprising given the packages have 22M downloads. Doesn't that warrant some kind of analysis on the potential impact on everyone currently depending on it?

ServiceStack references the now delisted Reflection.Emit packages in ServiceStack.Text which is the base dependency package used in effectively every ServiceStack NuGet package as well as several other NuGet packages using it as a dependency. We only provide netstandard2.0 and net45 builds, forcing targeting .netcore platform would break every single dependency and every netstandard2.0 project using it - i.e. the preferred target framework for creating cross-platform builds that support both .NET Framework and .NET Core.

So what's the recommendation now for packages using Reflection.Emit? Stop publishing .NET Standard 2.0 builds and tell everyone they can no longer create .NET Standard 2.0 builds for their libraries and projects?

The previous solution for using .NET Standard APIs that don't implement the API was to throw PlatformNotSupportedException runtime Exceptions, why exactly is that not the solution here?

[jbogard]

@seesharper yes, I added those but that might leave out others. The API docs list more but what others might that be missing? Is that a comprehensive list of possible TFMs that support the API? What about, say, xamarinxboxone?

[FransBouma]

I've refactored my DynamicMethod/ILGenerator code which I used to emit setter methods for properties at runtime with a Lambda.Compile() solution so the dependency on Reflection.Emit is now gone, however I had to spend 3-4 hours on it which I would have liked to have spend on other things. But alas, that's life when things 'move fast and break often' I guess?

Anyway, what I find a bit disturbing is the deafening silence from Microsoft personnel in this thread for the past weeks. Feels like debating things in a room when the people who actually can change things are not there.

I totally agree with @mythz and others here, it's poorly communicated, and with 22M downloads this was a silly decision with far-reaching consequences.

[jbogard]

@mythz I don't know how you're using it, but for my stuff, I had to revert back to #if feature flags to remove functionality in platforms that don't support building types on the fly (in my library, being able to map to an interface and I create a proxy on the fly).

[mythz]

The problem with this package is that it should never have been put on NuGet as a netstandard package since generating new types at runtime cannot be supported on full AOT platforms.

Disagree. The major .NET Core and .NET Framework platforms support it, just because AOT environments don't shouldn't preclude it being able to support the major .NET Platforms targeting netstandard2.0.

Has this now become the strategy for .NET Standard going forward? APIs not supported in AOT environments are now going to be removed? So there's going to be no more PlatformNotSupportedException Exceptions, the APIs are going to be yanked instead moving back towards a PCL subset? Or are we going to be inconsistent and selectively reduce the API surface for some features but continue to throw in others?

Supporting the intersection of the lowest common denominator APIs was what PCL did and resulted in a horrible development experience which forced the PCL bait and switch technique and delegating to multiple platform-specific implementations, which is all the investment we threw away when we moved to .NET Standard.

The best solution was the existing one (esp. as removing it now is a disruptive and breaking change to its transitive deps) which lets us use Reflection.Emit in all platforms that supported it whilst needing to implement a fallback for those that don't. Removing the ability to use Reflection.Emit in .NET Standard means libraries that use it can no longer target .NET Standard. What is even the point if we can't use features that .NET Core and .NET Framework share in a platform-agnostic abstraction? It would just be adding unnecessary abstractions/confusion/complexity to the .NET ecosystem with no benefits.

[weshaggard]

Thanks for the feedback everyone. Based on the overwhelming feedback we have relisted the latest version of the existing packages:

https://www.nuget.org/packages/System.Reflection.Emit/4.3.0
https://www.nuget.org/packages/System.Reflection.Emit.Lightweight/4.3.0

However as mentioned earlier in this thread these packages claim to be netstandard1.1 compatible but that is a lie. They will only work on .NET Core and .NET Framework. So if you are consuming them from a netstandard library expect that library to fail if it is ran on any other .NET Standard implementation.

We still don't have a great solution for supporting these on netstandard2.0 but the owners (@atsushikan @joshfree) of System.Relfection.Emit will be trying to figure out the longer term solution.

[seesharper]

@svick

As far as I know, Reflection.Emit (including Reflection.Emit.Lightweight) is primarily used for performance. But if you implemented Reflection.Emit.Lightweight using an IL interpreter, its performance would likely be pretty bad. So I'm not convinced the effort of supporting IL interpretation mode in Reflection.Emit.Lightweight would be justified.

It is my understanding that LambaExpression.Compile() and its execution is interpreted on AOT platforms and hence it should be no worse doing it for System.Reflection.Emit.LightWeight.
There is a lot of libraries using this package and having a "true" netstandard2.0 package would suddenly enable those packages to run on your iPhone without forcing the developers of these packages to rewrite their code to match the different platforms. The performance would be degraded, but it would work just as it does today with LambdaExpression.Compile()

@mythz

Has this now become the strategy for .NET Standard going forward? APIs not supported in AOT environments are now going to be removed? So there's going to be no more PlatformNotSupportedException Exceptions, the APIs are going to be yanked instead moving back towards a PCL subset? Or are we going to be inconsistent and selectively reduce the API surface for some features but continue to throw in others?

IMHO, throwing PlatformNotSupportedException is just another manifestation of the API subset we saw with PCL's. As the case is with System.Reflection.Emit.LightWeight, we can actually implement it and there is no reason to throw a PlatformNotSupportedException. I do however agree that it is problematic that less capable platforms restricts the standard from moving forward. But that is sort of the nature of having a standard. When additions are made to the standard, it must be ensured that it is at least somehow possible to implement by the platforms implementing the standard. At least among the major players and would say Xamarin fits that category.

This figure as it stands today is kind of a lie as I see it. Xamarin claims to support netstandard, but that support is riddled with exceptions which for us might seem reasonable. For the novice it might not be so easy to understand.

@weshaggard

Continuing to publish System.Reflection.Emit as a netstandard package is not a good solution in the long run. It makes the standard "false" again and should only be available for net/netcoreapp. It is a bitter pill to swallow, believe me I know. I have exact same issue with 'LightInject.Interception' that is a proxy library using System.Reflection.Emit. But I would rather target netcoreapp over lying to the consumer that this library can run everywhere.

[mythz]

@seesharper

IMHO, throwing PlatformNotSupportedException is just another manifestation of the API subset we saw with PCL's. As the case is with System.Reflection.Emit.LightWeight, we can actually implement it and there is no reason to throw a PlatformNotSupportedException.

They're not equivalent at all. The only reason why .NET Standard is useful is due to its vastly greater API surface which allows us to create a single build to run on multiple platforms. PCL's are far less useful as its reduced API intersection subset forced creation of multiple platform specific builds for non-trivial functionality. Their capabilities are not equivalent and the solutions they lead to are vastly different in complexity for both library authors and consumers, with .NET Standard we have a single build which runs on all platforms as we're able to check at runtime if the platform supports Reflection.Emit, if it doesn't we fallback to Lambda Expressions and Reflection. If these API's wasn't available to .NET Standard libraries we couldn't have a single build and would need to go back to developing and maintaining non-portable platform-specific builds.

Continuing to publish System.Reflection.Emit as a netstandard package is not a good solution in the long run. It makes the standard "false" again and should only be available for net/netcoreapp.

Forcing the creation and dependence of platform-specific builds is a far worse outcome, esp. with so many transitive dependencies already dependent on it. The primary benefit of .NET Standard is to be able to create libraries and projects to target a single useful abstraction, if we can't use it to access mainstream features available in .NET Core and .NET Framework it fails to resolve its primary use-case and may as well not exist since it's just adding more abstractions/confusion to the .NET ecosystem without any benefits over PCLs.

PCL's already tried binding to abstractions which only expose the intersection of API features available in each platform, which is effectively what you're asking to back to, since you'll only have access to APIs available in each platform and by extension only contain APIs available in the most restricted platform.

Having a broader API Surface in .NET Standard is far more useful and gives library authors the freedom of being able to choose how they want to handle support for different platforms, if these APIs weren't available that option wouldn't exist, it would force platform-specific builds and force all its dependent libraries and projects to also maintain platform specific builds, adding friction, fragmentation, reducing portability and limiting support to only the platforms authors choose to maintain builds for vs all the platforms supporting .NET Standard that support that feature.

It's not just a matter of preference, removing mainstream API support has far reaching disruptive impact on what solutions are available, forces unnecessary platform specific builds impacting the utility of libraries whilst breaking every .NET Standard library and project using them.

[davidfowl]

@mythz you’re absolutely right.

[terrajobst]

@masonwheeler

I completely understand that hiding the ref emit package is frustrating, considering the fall out that it caused. I also understand that us optimizing for execution environments that you don't care about can be upsetting, especially when this includes trade-offs that affect scenarios that you care about negatively.

That being said, I've marked your comment as "abusive" as it's not constructive. Insulting people isn't going to get these issues addressed. Also consider that this isn't a policy we control. The fact that you cannot generate and execute code at runtime on iOS is a policy decision by Apple. Also, in some cases it's not a policy restriction but a technological one.

There as three options with reflection emit:

1. Fix the current packages and make them work on top of .NET Standard 2.0. The issue is a weird inheritance relationship between Type and TypeInfo in .NET Standard 1.x compared to 2.0 and having non-public constructors. I can go into more detail, but in order to make the package actually buildable without restoring to hacks is a bit tricky, which is why we originally decided to hide the packages.

2. Add reflection emit to .NET Standard vNext. In order to support AOT-only scenarios this would likely include a capability API that allows consumers to check whether code generation is supported or efficient (i.e. whether we use real execution as opposed to interpretation).

3. Not exposing reflection emit to .NET Standard and require library authors to multi-target. This was basically the original path when we started hiding the package.

Right now, we're leaning towards the first option but the second option is also on the table.

[davidfowl]

I’m in favor of 2. It ubiquitous enough that it should be there. We need the capability check anyways for other reasons.

[terrajobst]

I’m in favor of 2. It ubiquitous enough that it should be there. We need the capability check anyways for other reasons.

So am I but it won't help anyone who is currently relying on .NET Standard 2.0 as this requires a new version of the standard. However, the APIs in question already exist.

So I'm thinking about a minimal fix to make this packages work as-is but also fix .NET Standard 2.0 vNext.

[bording]

@terrajobst Is there something about option 1 that makes option 2 harder to do? It seems to me that getting a version of the packages that target netstandard2.0 instead of needing to pull in the netstandard1.1 package graph would be a good short-term goal. Then for vNext of the standard, it could become part of it, and you wouldn't need the packages anymore.

So, unless there's a really good reason not to, I vote for both 1 and 2.

[terrajobst]

@bording

Looks like we posted virtually at the same time. Hopefully my comment addresses your question :-)

[aka-nse]

Please note that netstandard2.0 supports System.Reflection.Assembly.Load(byte[] rawAssembly) static method, which loads an assembly from binary image byte array. This means that it is possible to implement not all but most System.Reflection.Emit APIs' compatibles under pure netstandard2.0.

(Sorry, I don't know how many platforms support Assembly.Load without PNSE)

[ericstj]

@glassgrass what frameworks do you imagine a netstandard2.0 implementation would apply to?

I can certainly imagine some implementation that would write out the IL and produce an assembly like an API-driven ILASM, but I don't see current frameworks where I would want to use that.

Frameworks that have a JIT (and support Assembly.Load) could also support Ref.Emit directly, and provide their Ref.Emit implementation instead of a netstandard2.0 version. I know this is the case at least desktop/.netcore/.netnative. The first two support ref.emit and provide an implementation, the later does not, nor does it have a JIT nor support dynamic assembly loading.

I do think it would be an interesting project for someone to play around in corefxlab to write something on top of System.Reflection.Metadata that looks like Ref.Emit (or better). Not sure if anyone is looking at that already. /cc @nguerrera @tmat

[ghost]

I do think it would be an interesting project for someone to play around in corefxlab to write something on top of System.Reflection.Metadata that looks like Ref.Emit (or better).

That's on my plate: (https://github.com/dotnet/corefx/issues/4491 and https://github.com/dotnet/corefx/issues/2800)

https://github.com/dotnet/corefx/issues/2800 will come first as the ref emit stuff is likely to be an extension of it. I started prototyping 2800 this week and will be continuing that work in the months ahead. I'm not moving it to corefxlab until I'm much further along. Workflow is too inefficient there for my tastes.

[stakx]

@atsushikan: Glad to see that there is still ongoing work on this issue. Thanks for tackling this.

I've mentioned the following elsewhere previously: One thing that System.Reflection.Metadata and Assembly.Load(byte[]) cannot do is incremental type-by-type emitting. You can't emit one type, then another... you can only ever emit complete assemblies, which means this is not too useful for e.g. mocking libraries that rely on dynamic proxy type generation.

Of course one could only ever produce single-type assemblies, but this becomes potentially inefficient & difficult when making internals visible to the dynamically generated assemblies via [assembly: InternalsVisibleTo] (another thing mocking / testing libraries often require). You'd have to be able to give several dynamic single-type assemblies the same identity / strong name for this to work. (The runtime might already allow this in practice, but I can't find documentation officially confirming it.)

[tmat]

@stakx You can emit System.Runtime.CompilerServices.IgnoresAccessChecksToAttribute to the dynamic assembly to allow it to access non-public members of specified assembly, instead of using IVT.

[stakx]

@tmat: This is interesting. Is this attribute officially documented somewhere? I can't seem to find any information about it. Never mind, I think I've found enough information about this.

It's a pity this special attribute isn't officially documented, sort of makes it a bit of a gamble to use it outside the runtime.

[abhi18av]

Would this be added to .NET core 3.0 ?

[karelz]

cc @steveharter @joshfree

[joshfree]

@karelz https://github.com/dotnet/corefx/issues/30654 tracks the work for 3.0

[karelz]

@joshfree is this one a dupe, or does it cover more?

[joshfree]

@karelz it covered the issue with the Ref.Emit package being incorrectly delisted from nuget and then the package being re-listed by @weshaggard. I think this issue can now be closed as the remaining work is tracked with dotnet/corefx#30654.

[karelz]

OK, closing then :) ... the remaining work is tracked in dotnet/corefx#30654 as mentioned above.