Access denied for NuGet package source while build dotnet/cli 3.1.300 Tag #11795
Comments
|
Any chance I can get an update on this? This is blocking us rolling out the update at AWS. |
|
I'm having this problem now as well. I'm trying to create a Homebrew formula for .NET Core (see #4600) and in order to do so, I have to be able to build .NET Core from source. With the required private NuGet package feeds, it currently seems impossible to build .NET Core from source. Is this intentional? |
|
Seems like dotnet/cli@7c1fa3d#diff-a7c26590e27fcbb61c69ff2564beb1fa changed the feeds from Simply removing the internal feeds from Trying to build a version from before dotnet/cli@7c1fa3d, such as v3.1.201, ends up printing an almost endless amount of the similar error message to the console: Finally failing and halting on the following error: Trying to access one of the faulty feeds ends up returning the following JSON-wrapped exception: {
"$id": "1",
"innerException": null,
"message": "Can't find the package 'runtime.native.system' in feed 'dotnet3.1'.",
"typeName": "Microsoft.VisualStudio.Services.NuGet.WebApi.Exceptions.PackageNotFoundException, Microsoft.VisualStudio.Services.NuGet.WebApi",
"typeKey": "PackageNotFoundException",
"errorCode": 0,
"eventId": 3000
}So currently, it seems impossible to build any version of .NET Core from source unless you're hired by Microsoft and have access to their internal NuGet feeds. That's very, very unfortunate. A fix would be highly appreciated. 🙏 |
This has to do with how 3.1 dependency update works... we decided to go with a simpler approach (in some respects) where every single dependency update is checked into the repo. This has an unfortunate chain of dependencies leading to the tooling checking in each internal NuGet feed required:
The intent is that on release day, every single internal-only package is published to the public feeds. Your attempt to fix by removing the internal feeds from the nuget.config should have worked! (IMO it isn't documented well enough, we need to make this obvious.) @mmitche has dotnet/arcade#5233 opened, to track having the build script automatically remove the internal feeds so you don't need to do this step manually. dotnet/source-build actually relies on this exact fix working. For example, https://github.com/dotnet/source-build/tree/v3.1.106-SDK was an internal build that we can now build from source publicly. The dotnet/source-build infrastructure dynamically deletes internal feeds from each repo's NuGet.config before building it. (For context: dotnet/source-build is how Fedora got its fully built from source 3.1.106 SDK. dotnet/source-build also handles building the .NET repos in the correct order and passing dependencies through, so it's more truly "from source" than just building from an individual repo's tag and letting it download Microsoft-built binaries. Note that it only maintains 3.1.1xx, not 3.1.2xx, so switching over might not work for everyone, but I would recommend it if you can.) I'm not sure why that the nuget.config fixup didn't work in this case. Someone familiar with dotnet/sdk in particular (or the dotnet/arcade toolset) needs to look into that, and I think this issue tracks that adequately. @marcpopMSFT |
|
There is another option:
The downside of SetupNuGetSources.ps1 is:
|
|
Alternatively, you may be able to remove the internal darc feeds from the NuGet config. @mmitche is going to look for any documentation arcade might have on this process. A quick summary is that when we identify a security vulnerability we need to fix, we make the fix and produce builds for testing the fix internally. This limits the risk of exposure as we don't want to be public with the specific security fix until we have the fix signed off and ready for consumption. Arcade doesn't allow us to maintain parallel public and internal channels for flowing code so our public builds during these times can end up in an odd state. We'll try to get some documentation updated to help unblock you for how to build as typically we're only in this state for a short period of time as we also want to get the security fixes out as quickly as we can. |
This was tried and and it failed, see above--in particular #11795 (comment). Can you look into it (or otherwise triage this issue)? |
|
Ahh I missed that line there. Trying locally on my machine. |
|
@asbjornu When you get the errors after removing the darc-int feeds...are you building from the command line, VS, etc. I eliminated them without any issues and built on the command line. In fact, those two (v3.1.300) feeds no longer contain any packages at all. Every night we run through every one of our feeds and eliminate those packages that also exist at the same version on nuget.org. This keeps nuget.org as the "source of truth". |
|
We saw the same issue in source-build ( @asbjornu where did you get the source code from? |
|
I was building in iTerm2, running |
Thanks for this info, might be useful to know to justify some things.
For now, having (At least, that works for me on a downloaded 3.1.302 source zip with the nuget.config modification. (I also got rid of the extra runtime installs in |
|
Thanks for that @dagood. With your instructions, I've now managed to build Building I was able to get much further with Ideas? |
This is an MSBuild bug with niceness level I happened to hear about at dotnet/core#4719. I'm not sure what a simple workaround is here, but you may need to adjust niceness level up or down to match what MSBuild expects.
Seems unrelated to me, worth opening an issue about.
This seems like a really weird limitation of that build system to me. Why is starting a build with |
|
What I did to get around the issue was remove the internal darc feeds and the build worked fine for me afterwards. I would really like a better fix then me making changes to the repo before doing a build. |
For sure. To reiterate, that's tracked by dotnet/arcade#5233 (or an offshoot issue from there in the future). |
|
To me, it feels like using the Building the |
|
Thanks to dotnet/source-build#1695 I'm now able to start the build from the provided It looks quite suspicious that version Before I file an issue in the massively overwhelmed AspNetCore repository, do you have any ideas or input, @dagood? |
|
This can have a variety of causes and we've run into (and fixed) a few in the past caused by dependency flow and source-build tweaks. That 3.1.0.0 might be fine and intended to discover a 3.1.6.0 DLL that's just not present. You could try using ilspy, dnspy or a similar tool on the DLL that failed to load and DLLs in the dirs being referenced to try to figure out the state of the build, but that's only the first step. @asbjornu can you file an issue in dotnet/source-build? I suspect this is something about our infra. The macOS CI build is working in source-build (a smoke-test fails causing overall failure, but the build is ok), so there's probably some difference caused by configuration/env we'll need to figure out. Check out https://aka.ms/binlog first for the caveats (env is included). |
|
After some progress in Homebrew/homebrew-core#53092 (comment), I'm now able to successfully build with the following Homebrew formula. The missing part is installing the built tarball. I hope to get that resolved with the help of the Homebrew team over the next few days. The only question I'm left with here is: What version should we give the formula? Is the inferred version from the Git tag ( |
Using the SDK version is correct:
We have some info about packaging recommendations at https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging, but it actually gets this specific piece of info wrong so I've filed dotnet/docs#19921 to fix it. |
|
Thanks, @dagood. SDK version it is. I now have an almost working formula. The outstanding issues are related to .NET's environment requirements as per Homebrew/homebrew-core#53092 (comment):
These requirements make finalizing the installation very difficult. Point 1 is impossible to do in a Homebrew formula since Homebrew doesn't require (or even ask for) What we would have to do in Homebrew is write detailed instructions on what a user needs to do to create the What's worse is that even with the files and enviornment variables in place, the LD_LIBRARY_PATH="/usr/local/lib" dotnet --info
Process terminated. Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
at System.Environment.FailFast(System.String)
at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
at System.Globalization.GlobalizationMode..cctor()
at System.Globalization.CultureData.CreateCultureWithInvariantData()
at System.Globalization.CultureData.get_Invariant()
at System.Globalization.CultureInfo..cctor()
at System.Version.TryParseComponent(System.ReadOnlySpan`1<Char>, System.String, Boolean, Int32 ByRef)
at System.Version.ParseVersion(System.ReadOnlySpan`1<Char>, Boolean)
at System.Version.TryParse(System.String, System.Version ByRef)
at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetDarwinVersion()
at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetOSVersion()
at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment..cctor()
at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment.GetRuntimeIdentifier()
at Microsoft.DotNet.Cli.MulticoreJitProfilePathCalculator.CalculateProfileRootPath()
at Microsoft.DotNet.Cli.MulticoreJitActivator.StartCliProfileOptimization()
at Microsoft.DotNet.Cli.MulticoreJitActivator.TryActivateMulticoreJit()
at Microsoft.DotNet.Cli.Program.Main(System.String[])
Abort trap: 6So before figuring out what to write in a LD_LIBRARY_PATH="/usr/local/Cellar/icu4c/67.1/lib/" dotnet --infoIf we can get around this ICU problem, it would be interesting to see whether the other requirements are actual requirements or just recommendations. |
|
Seen some build servers have an executable bash script in path named |
Some more detail about these locations: https://github.com/dotnet/designs/blob/92af9e911006df64cfe75714121584e1b2d21ee9/accepted/2020/install-locations.md It sounds like you aren't making what would typically be considered a distro package, so yeah, don't worry about that. With these limitations it sounds like what you need is a per-user install. Some more detail here: (Points to https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#manual-install for details on per-user install.)
This is a known issue with the dotnet/coreclr repo on new versions of CMake, which has had a fix committed for the next release. (I linked you this earlier and I thought it had something to do with Big Sur, but it's just CMake.) More details and getting the fix in a dotnet/source-build release is tracked by dotnet/source-build#1685. dotnet/coreclr#28066 looks like a simple enough change to add a patch for, for now. This is getting fairly long and off the original topic, I'd recommend filing these types of issues in dotnet/source-build. |
It looks like this is actually pretty common in Homebrew with |
|
Thanks for the tip on |
Replied at dotnet/source-build#1685 (comment). |
|
With Homebrew/homebrew-core#60929 merged, this can be considered fixed for me at least. |
Open Mac Settings
enable vs code and vs studio
And For Terminal1- cd /Users/your_user_name/.local/share/ |
|
old issue about building 3.1.300. Closing. Please file a new issue if this still affects 7.0.100. |
I'm trying to build version 3.1.300 of the dotnet/cli repository but the
build.shfails during the restore process with the following error message:When I try to go directly to the URL for the package source I get access denied:
The
https://pkgs.dev.azure.com/dnceng/_packaging/darc-int-dotnet-core-setup-0c2e69ca/nuget/v3/index.jsonpackage source is coming from the NuGet.config file here https://github.com/dotnet/cli/blob/v3.1.300/NuGet.configWhy are the package source requiring authentication and how are automation machines not running in the .NET team's build system supposed to be able to build the project if authentication is required.
The text was updated successfully, but these errors were encountered: