Mac arm64 binaries are bad because the pipeline installs x64 .NET

[jasongin]

Opening a separate issue to track this, after discussion in comments of #56.

The root cause is a bug in the .NET install task that the Azure DevOps build pipeline uses. We should be able to work around that by using a different method to install .NET, such as the install shell script.

[spacepluk]

I was trying to use this to call some C# code from nodejs. The objective is to deploy this in a linux container but I'm developing on a Mac with an arm cpu.

I replaced the binaries in the npm package with some I built myself and I managed to call System.Console.WriteLine. But then when I was trying to build a C# library with some stuff annotated with [JSExport] the binaries in the microsoft.javascript.nodeapi.generator package seem to be Windows only.

I hope that helps.

[jasongin]

the microsoft.javascript.nodeapi.generator package seem to be Windows only

That package is supposed to be platform-independent, as it is an MSIL assembly. But due to this issue we haven't been able to test on Mac arm64 so maybe it's not correctly built to be fully platform-independent.

[spacepluk]

This is what I'm getting in my computer:

[spacepluk]

And this is the error I get when I try to build a csharp library with the NodeApi packages:

MSBuild version 17.3.2+561848881 for .NET
  Determining projects to restore...
  Restored /Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj (in 97 ms).
/Users/spacepluk/.dotnet/sdk/6.0.417/Microsoft.Common.CurrentVersion.targets(2302,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi/0.4.26/lib/net6.0/Microsoft.JavaScript.NodeApi.dll", "AMD64". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
/Users/spacepluk/.dotnet/sdk/6.0.417/Microsoft.Common.CurrentVersion.targets(2302,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/lib/net6.0/Microsoft.JavaScript.NodeApi.Generator.dll", "AMD64". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
CSC : warning CS8034: Unable to load Analyzer assembly /Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll : Could not load file or assembly 'Microsoft.JavaScript.NodeApi.Generator, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null, processorArchitecture=AMD64'. The given assembly name or codebase was invalid. (0x80131047) [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
  HeroGenJs -> /Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/bin/Debug/net6.0/HeroGenJs.dll
  Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'. An attempt was made to load a program with an incorrect format.

  File name: '/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'
  /var/folders/__/x_hjp9c93cg4ch1jbvsykm240000gn/T/MSBuildTempspacepluk/tmp8cf02ea4e42a4ab4af5dc1acb226c7b6.exec.cmd: line 2: 72989 Abort trap: 6           dotnet "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Debug/net6.0/Microsoft.JavaScript.NodeApi.Generator.rsp"
/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/build/Microsoft.JavaScript.NodeApi.Generator.targets(47,5): error MSB3073: The command "dotnet "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Debug/net6.0/Microsoft.JavaScript.NodeApi.Generator.rsp"" exited with code 134. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]

Build FAILED.

/Users/spacepluk/.dotnet/sdk/6.0.417/Microsoft.Common.CurrentVersion.targets(2302,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi/0.4.26/lib/net6.0/Microsoft.JavaScript.NodeApi.dll", "AMD64". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
/Users/spacepluk/.dotnet/sdk/6.0.417/Microsoft.Common.CurrentVersion.targets(2302,5): warning MSB3270: There was a mismatch between the processor architecture of the project being built "MSIL" and the processor architecture of the reference "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/lib/net6.0/Microsoft.JavaScript.NodeApi.Generator.dll", "AMD64". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and references, or take a dependency on references with a processor architecture that matches the targeted processor architecture of your project. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
CSC : warning CS8034: Unable to load Analyzer assembly /Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll : Could not load file or assembly 'Microsoft.JavaScript.NodeApi.Generator, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null, processorArchitecture=AMD64'. The given assembly name or codebase was invalid. (0x80131047) [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/build/Microsoft.JavaScript.NodeApi.Generator.targets(47,5): error MSB3073: The command "dotnet "/Users/spacepluk/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.26/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Debug/net6.0/Microsoft.JavaScript.NodeApi.Generator.rsp"" exited with code 134. [/Users/spacepluk/Hacking/company/project/express_server_postgres_testing/HeroGenJs/HeroGenJs.csproj]
    3 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.42

[spacepluk]

Happy to test things if it helps.

[jasongin]

Update: I found the right people on the Azure DevOps team and got them to look at this issue. A fix for the pipeline task bug was merged yesterday. The release process for the new pipeline task version will take a couple weeks. After that we should be able to just update our pipeline task and finally have a good Mac arm64 build.

[luanmm]

Just to contribute (I think/hope): I'm getting the error below when using this package in a Dockerfile that targets arm64.

Could be related to this issue? Or could be something else?

I'm not sure what is causing this (the error message seems to be misleading).

[jasongin]

@luanmm Is the Docker container OS Linux? This issue is specifically about arm64 on Mac OS. It doesn't affect arm64 builds for Linux.

Could not load file or assembly 'Microsoft.JavaScript.NodeApi, ...'. The system cannot find the file specified.

Obvious question, but have you checked whether the file Microsoft.JavaScript.NodeApi.dll exists? If you get it from npm install it should be under node_modules/node-api-dotnet/net6.0.

[luanmm]

I'm using a Linux host on amd64 targetting a Linux on arm64.

This is a simple use case, only in .NET/C# (I'm not working with interop from the JS side). But indeed, the dll seems not to be produced actually.

The "misleading error" that I stated is because I would expect that an error that is not producing the output should be thrown before (in the build, for instance). I could not understand why it happened. I can dig a little deeper if needed/helpful.

[luanmm]

Oh, now I found this: #80. Perhaps the problem is indeed the Linux arm64 version not being published yet in Nuget package? 😄

If so, just sorry for mixing up the subjects here!

[jasongin]

I'm actually not sure about the status of Linux arm64 builds. Regardless, the error you reported is related to a platform-neutral (MSIL) assembly file, so it might not be related to the arm64 platform anyway.

sorry for mixing up the subjects here!

Please open a separate issue if you want to continue troubleshooting. And if you do, include more information about how your project is organized, built, packaged, and deployed.

[sumeet-singh04]

Update: I found the right people on the Azure DevOps team and got them to look at this issue. A fix for the pipeline task bug was merged yesterday. The release process for the new pipeline task version will take a couple weeks. After that we should be able to just update our pipeline task and finally have a good Mac arm64 build.

@jasongin @vmoroz It seems like the Azure pipeline bug was merged a couple of weeks ago. Were the OSX ARM64 binaries fixed for node-api-dotnet ?

[sumeet-singh04]

@jasongin @vmoroz Thanks for fixing the build, I tried using version 04.34 on mac osx ARM64, but got the below error.

node-cblite -> /Users/sumeetsingh/development/ngtos/node-cblite/bin/Release/net6.0/osx-arm64/node-cblite.dll
  Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'. An attempt was made to load a program with an incorrect format.
  
  File name: '/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'
  /var/folders/wj/yl2ysnj11ybbgjfl_qwdr2ym0000gq/T/MSBuildTempsumeetsingh/tmp2abb273f6bce42f992b445886b379fdd.exec.cmd: line 2: 27522 Abort trap: 6           dotnet "/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Release/net6.0/osx-arm64/Microsoft.JavaScript.NodeApi.Generator.rsp"
/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/Microsoft.JavaScript.NodeApi.Generator.targets(47,5): error MSB3073: The command "dotnet "/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Release/net6.0/osx-arm64/Microsoft.JavaScript.NodeApi.Generator.rsp"" exited with code 134. [/Users/sumeetsingh/development/ngtos/node-cblite/node-cblite.csproj::TargetFramework=net6.0]

[coding-velociraptor]

@jasongin @vmoroz Thanks for fixing the build, I tried using version 04.34 on mac osx ARM64, but got the below error.

node-cblite -> /Users/sumeetsingh/development/ngtos/node-cblite/bin/Release/net6.0/osx-arm64/node-cblite.dll
  Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'. An attempt was made to load a program with an incorrect format.
  
  File name: '/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll'
  /var/folders/wj/yl2ysnj11ybbgjfl_qwdr2ym0000gq/T/MSBuildTempsumeetsingh/tmp2abb273f6bce42f992b445886b379fdd.exec.cmd: line 2: 27522 Abort trap: 6           dotnet "/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Release/net6.0/osx-arm64/Microsoft.JavaScript.NodeApi.Generator.rsp"
/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/Microsoft.JavaScript.NodeApi.Generator.targets(47,5): error MSB3073: The command "dotnet "/Users/sumeetsingh/.nuget/packages/microsoft.javascript.nodeapi.generator/0.4.34/build/../analyzers/dotnet/cs/Microsoft.JavaScript.NodeApi.Generator.dll" "@obj/Release/net6.0/osx-arm64/Microsoft.JavaScript.NodeApi.Generator.rsp"" exited with code 134. [/Users/sumeetsingh/development/ngtos/node-cblite/node-cblite.csproj::TargetFramework=net6.0]

It looks like related to the missing dlls for other architectures on NuGet. If you reference the local dlls of the NPM package, you migh get further.

[sumeet-singh04]

@coding-velociraptor Yes, when I built the library on my local using dotnet pack and used the generated node-api-dotnet npm package and NUGET package in my project, it worked. So that means it is still related to the build process

[luanmm]

This is really looking the same scenario as I'm having here when building my project for ARM64 (even though I'm on a Linux OS) - I'm developing in an AMD64, but publishing the project in an AWS Graviton (which uses ARM64 arch). Locally, the Nuget package works (in an AMD scenario), but in AWS it is broken (ARM scenario).

My workaround, for now, is doing a clone in the node-api-dotnet project and referencing/building it locally. Seems to be something with the building/packaging process, indeed, for some reason.

[jasongin]

We know the published Mac arm64 build still isn't functional. @vmoroz recently updated the build agent to arm64, but that triggered another problem.

[jasongin]

@vmoroz Do you know what the status of this is? Are the published binaries working now on Mac arm64, after fixing #222?

[jasongin]

Fixed in v0.5.10.