NuGetAuthenticate and on-premise (Azure DevOps Server 2019)

[chucker]

Type: Feature

Enter Task Name: NuGetAuthenticate

Environment

• Server - Azure DevOps Server 2019 Update 1.1

• Agent - Private, Windows Server 2012 R2, agent 2.153.1

Issue Description

I have a project with Azure Pipelines, Azure Code (git), and Azure Artifacts (NuGet) to store some private packages. Trying to access that artifact from the pipeline seems to be rather cumbersome in terms of authentication.

Recent guidance such as here suggests to use the NuGetAuthenticate task.

However, that task seems to only be available on the cloud version. (the docs seem to support this: they don't list it for "Azure DevOps Server 2019".)

Given the usefulness, is porting the task to on-premise planned? Can I manually install the task somehow, or is there a reasonable alternative?

[satbai]

Hello @chucker
You are correct, the NuGet Authenticate task doesn't exist in the 2019 version of Azure DevOps Server, but it is the latest recommended way to authenticate to Artifacts feeds from pipelines (and exists in 2020+).

We have a script that you can run that will add the auth task to your on premise instance. However, for now we haven't officially published the scripts yet because we haven't tested the scripts behind a proxy. Is your Azure DevOps Server behind a proxy? Before running one of the scripts, please read through it to understand what it does. Here's the script for PowerShell (Windows) and here's the one for Bash (Linux/Mac). Here are the main steps the script takes:

• install tfx-cli
• git clone azure-pipelines-tasks to a temporary directory
• 'npm install' and run build on preferred tasks
• upload task(s) to your Azure DevOps Server instance using tfx-cli (docs related to the pipeline task update with tfx-cli)
• remove the created temporary directory

Setup before running the script. For both these scripts you will need:

• Collection url
• PAT token
• (If you need proxy setup) Proxy information (url, username, password)

Powershell script:
Open up a powershell terminal and run the following command:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/azure-pipelines-tasks/36c1ff90870b4363d6d20525df85c1409ffc748b/Tasks/Common/packaging-common/auth-tasks-to-azure-devops-server.ps1) } "

Provide the needed parameters. By default this script adds NuGet Authenticate, Maven Authenticate, Pip Authenticate and Twine Authenticate tasks. If you only want to add the NuGet Authenticate task you can run the script with a "-Task NuGetAuthenticateV0".

Shell / Bash script:
Open up a bash terminal and run the following command:
bash <(wget -qO- https://raw.githubusercontent.com/microsoft/azure-pipelines-tasks/users/aamallad/build-tasks-script/Tasks/Common/packaging-common/auth-tasks-to-azure-devops-server.sh) --token your-token --collection-url your-collection-url

If the script ran successfully you can now add the auth task to your pipeline. Let me know if you have any questions or if you run into any issues.

Edit: Please note that you must run the script in a directory with a short path such as "C:\MyTest", otherwise the step where NuGetAuthenticate gets built may fail when the build extracts the artifacts credential provider.

Edit 2: Azure DevOps Server 2020+ includes the NuGet Authenticate task.

[chucker]

Hi @satbai, and thank you for your response.

I'm not behind a proxy. I'll take a look!

[harleydk]

Hi @satbai, and thank you for your response.

Thanks @chucker for bringing it up, was in dire need of this, too.

[harleydk]

Hi there @satbai , I tried to follow your instructions and was succesfully able to install the tasks for maven, pip and twine. However, the NuGet task, that I so dire need, did not work. Enclosing screenshot - can you diagnose the issue? Thanks a million in advance.

[satbai]

I was able to reproduce your issue when I was running the script in a directory with a long path. However, when running it in a directory with shorter path it succeeded. E.g. "C:\MyTest". Sorry for the inconvenience! I'll include this in the instructions above.

[harleydk]

@satbai Terrific, and great service, really appreciate it.

[harleydk]

Hate to bring this up again, sorry, alas a shorter path didn't quiet do it for me:

[satbai]

The script is failing when it does "node make.js build --task NuGetAuthenticateV0".

And in the pipeline repo's build it looks like it's failing when it tries to download the credential provider. I think this open issue is the cause of this. Unfortunately the workaround there seems to be to downgrade node :(

[chucker]

@satbai just FYI, your Unix script won't (easily) work on macOS, as it only ships with bash 3.2.

/dev/fd/11: line 12: declare: -A: invalid option

My guess is that's a newer bash feature.

In Windows on PowerShell, I get:

error: Error: Invalid service url - path is too long. A service URL should include the account/application URL and the collection, e.g. https://fabrikam.visualstudio.com/DefaultCollection or http://tfs-server:8080/tfs/DefaultCollection

I've tried both https://mytfsserver.ourcompany.example/ProjectCollection/ and https://mytfsserver.ourcompany.example/ProjectCollection (no trailing slash). I get the same error either way. There's no space or special character in the project collection, so that's not the reason.

[satbai]

@chucker thanks for feedback! Interesting - I will try to get a repro with a long collection URL.

@aasim to comment on the bash script.

[harleydk]

Can confirm that using node version 11.15 (found here) worked, thanks @satbai for linking to the issue.

[Skoucail]

Thanks for this work around.
I wasted like half a day just to fix my dotnet nuget push... But this finally fixed it.
Just a little question.
Now i manually uploaded the task, if it ever gets released will this give issues?

[AndreFo]

I have used node version 11.15 and successfully uploaded these task to an on-premise devops server. Using tfx build tasks --list the NuGetAuthenticate task appears with following entry:

id            : f5fd8599-ccfa-4d6e-b965-4d14bed7097b
name          : NuGetAuthenticate
friendly name : NuGet authenticate
visibility    : 
description   : Configure NuGet tools to authenticate with Azure Artifacts and other NuGet repositories. Requires NuGet >= 4.8.5385, dotnet >= 2.1.400, or MSBuild >= 15.8.166.59604
version       : 0.167.1

However, it does not work to use it in a pipeline with

steps:
  - task: NuGetAuthenticate@0

The pipeline cannot be started. Azure DevOps Server complains that the task is missing.

Any hint, what I could have missed here?

Edit: The task can be added for each collection of the Azure Dev Ops server separately, so it was necessary to add it to the correct collection, to get it working. This is what I missed...

[dpurge]

I had the similar problem to solve, and satbai's comment above was super helpful to solve it.

What is needed is two commands (once you have tfx-cli installed):

node make.js build --task NuGetAuthenticateV0

tfx build tasks upload --task-path _build\Tasks\NuGetAuthenticateV0 --service-url $ServiceURL --token $PAT

I did not have time to fight with the error when downloading NugetCredProvider 0.1.20 that is listed in the archivePackages (not a nodejs developer), so I just dowloaded it in a browser, unzipped and patched make-util.js line 617:

//var archiveSource = downloadArchive(archive.url);
var archiveSource = 'C:\tmp\NugetCredProvider.0.1.20';

You could modify the downloadArchive() function to only download archives that have not been downloaded to a local directory-cache yet.

Or you could just zip and attach prebuilt tasks here which we could upload to on-prem tfs.

[detilium]

I'm getting the following error:

Failed to run 'npm install -g tfx-cli --registry=https://registry.npmjs.org/'
. You may have to manually install tfx-cli to run this script.

I tried to install tfx-cli manually, but that triggers a deprecation error:

npm : npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. P
lease, upgrade your dependencies to the actual version of core-js@3.

[steve-kuennen]

I could not get past the error downloading the cred provider (even after reverting to node 11.15) so I went with @dpurge and manually downloaded it. This got me past the node make.js build step (with the same hack to line 617)

When I attempt to run the next step 'tfx build tasks upload...' I get:

error: TypeError: Cannot read property 'value' of null

Oddly, this is the same error that led me to this post in the first place. I originally just omitted the NuGetAuthenticate task in my build pipeline, seeing as how it did not exist, and attempted to do a nuget push directly to my artifact feed which resulted in this exact error.

I'll continue looking at it tomorrow, but if anyone has come across this i'd appreciate some help. Been fighting with on prem artifact feeds for a few days now.

[a688]

@steve-kuennen I got around the download error by updating sync-request to the latest version. Even though I finally got the authenticate task installed on our DevOps Server instance, I still can't get my build process (or when I run dotnet restore manually on our build servers) to get credentials.

[dpurge]

@a688 I believe the way it works requires a few things to match. I use this task to set up credential provider to upload and download tasks to Azure Artifacts, and this is a process that I use during troubleshooting.

1. Create PAT in Azure Artifacts - I create it manually on https://dev.azure.com/dgsit/_usersSettings/tokens making sure that it works for my target organization (you can have more than one), taking note of the expiry date (maximum 1 year) and has the correct setting in "Packaging" section

2. Test the PAT - I do it from my desktop and commandline nuget.exe, using recent nuget version (I believe credential providers only work with recent versions of nuget.exe). I add the PAT to my profile nuget.config with nuget sources add -Name <FEED_NAME> -Source <AZURE_ARTIFACTS_FEED_URL> -username <USERNAME> -password <TOKEN> and then test listing, installing and pushing packages with nuget.exe and matching Source parameter.
   It has to work, which proves that PAT is working.

3. Set up the service connection in the project in your TFS. I am using nuget service, "External Azure DevOps Server" option, feed url that I copied from Azure Artifacts and my tested PAT. It will only work if this is set up all this correctly. Otherwise task will not fail, but credential provider will not have working credentials to return.

4. In the build definition, I set up NugetAuthenticate task, and make sure that when the build runs, the step with this task is green and in the log I see the message that the credential provider has been set for my service connection.

5. Only then do I test using the feed in the build. It is important that the feed url is matching perfectly the url you have in your service connection. If we have nuget.config in our project, I check that the url is maching perfectly. If it does not match, it will not work. I tend to use Nuget task to do nuget operations, but we also use nuget.exe from command line and it works. The only thing I know of that does not work is powershell module installation. Also, there is some trickery if you want to push your package from command line - my colleague has found that it only works for him if he passes "AzureArtifacts" as ApiKey. This seems to be a magic value.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days