Artifacts do not download on multi stage yaml build using DownloadPipelineArtifactV2

[giggio]

Required Information

Question, Bug, or Feature?
Type: Bug

Enter Task Name: DownloadPipelineArtifactV2

Environment

• Server - Azure Pipelines

  • account name: lambda3
  • team project name: sdk
  • build definition name: template
  • build number: buildid: 12494

• Agent - Hosted:

  • agent queue name: Default
  • OS of the machine running the agent and the agent version: Windows, agent version 2.155.1

Issue Description

When you try to download the artifacts on a multi stage build definition using the DownloadPipelineArtifactV2 task it will only download the artifacts from the first stage. In my scenario I have 3 stages, and I product the artifacts on the second stage, while the first stage only has a code coverage artifact. What I've seen is that the code coverage artifacts (1st stage) download, while the ones from the 2nd stage don't.

Note: I am using a self hosted agent, and I noted that the artifacts were not being removed between each build, which created problems, as leftover artifacts from previous builds were still around. So, I had the idea of downloading the artifacts to the Agent.TempDirectory, so it would be removed for sure.

This was all on a deployment stage.

If you let the download happen naturally, it will download all the artifacts, the problem only happens when you use the DownloadPipelineArtifactV2 task directly.

I also noticed that the Build Summary shows no artifacts for the second stage, it only shows one artifact for the 1st stage. But the artifacts tab does show all 3 artifacts.

Task logs

These are the partial logs for the download of the last two artifacts, named vsix and nupkg:

Information, Start downloading artifact - vsix
Information, Minimatch patterns: [**]
Information, Filtered 1 files from the Minimatch filters supplied.
Information, Could not initialize dataport.
Information, Downloaded 0.0 MB out of 0.1 MB (0%).
Information, Downloaded 0.1 MB out of 0.1 MB (100%).
Information, 
Download statistics:
Total Content: 0.1 MB
Physical Content Downloaded: 0.1 MB
Compression Saved: 0.0 MB
Local Caching Saved: 0.0 MB
Chunks Downloaded: 1
Nodes Downloaded: 0

Information, Start downloading artifact - nupkgs
Information, Minimatch patterns: [**]
Information, Filtered 1 files from the Minimatch filters supplied.
Information, Could not initialize dataport.
Information, Downloaded 0.0 MB out of 0.0 MB (0%).
Information, Downloaded 0.0 MB out of 0.0 MB (100%).
Information, 
Download statistics:
Total Content: 0.0 MB
Physical Content Downloaded: 0.0 MB
Compression Saved: 0.0 MB
Local Caching Saved: 0.0 MB
Chunks Downloaded: 1
Nodes Downloaded: 0

[TabithaLarkin]

I'm seeing what I can only imagine is the same problem.

- download: current
  displayName: 'Download Scripts'
  artifact: Topic
  patterns: '@(SetBranchName|StopAppPool).ps1'

In this case the new version of StopAppPool.ps1 included a bug that was picked up in code review, but somehow succeeded in the build pipeline. I determined that this was because the file had already been downloaded by a previous run to the agent, and when downloaded again it was not replaced.

This seems like a fairly serious problem if older files could be run or referenced by newer builds.

Logs:

Download from the specified build: #31512
Download artifact to: D:\_work\2/Topic
Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session dd19eddf-ef99-4fe8-a1d0-95af501bf76f
Information, Minimatch patterns: [@(SetBranchName|StopAppPool).ps1]
Information, ArtifactHttpRetryMessageHandler.SendAsync: [url] attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False
Information, Filtered 2 files from the Minimatch filters supplied.
Information, Could not initialize dataport.
Information, Downloaded 0.0 MB out of 0.0 MB (0%).
Information, Downloaded 0.0 MB out of 0.0 MB (100%).
Information, 
Download statistics:
Total Content: 0.0 MB
Physical Content Downloaded: 0.0 MB
Compression Saved: 0.0 MB
Local Caching Saved: 0.0 MB
Chunks Downloaded: 2
Nodes Downloaded: 0

[jahsu-MSFT]

@giggio I cannot repro the issue you mentioned. In my pipeline, I have stage 1 with build and pipeline artifacts published, in the stage 2, I have another build and pipeline artifact published and in the stage 3, I have a download pipeline artifact V2 task and it will download both build artifacts and pipeline artifacts if the multi-download mode is selected.
One thing to note is that, The jobs by default does NOT run in order, you will need to select dependencies. For in example, in this case, in the stage 3, I need to specify that the stage 1 and 2 as the dependencies in the stage 3.

Can you please let me know if you're still seeing the same issue after the dependencies are specified?

[GarethOates]

I've been experiencing this problem today as well. If an artifact has previously been downloaded by the agent, it is used in the subsequent stages instead of the latest being downloaded. Even though when you view the artifact in Azure DevOps, it shows the files as they should be.

In my case, I had renamed 2 files. The artifact in Azure DevOps looked ok, with the 2 new filenames being present, but when the job was running, it was using the older artifact version with the 2 old filenames. The solution for me was to log into each agent's work folder and delete the cached version.

I'm using the download keyword as part of a deployment job, and not the task.

- download: current
  artifact: <name-here>

This happened irrespective of whether or not I created a new Release.

[jahsu-MSFT]

@giggio Are you still seeing the issue after adding dependencies? If so, can you please provide your sample build def/yaml schema?

@TabithaLarkin @giggio
Regarding the old artifacts not being deleted in the agent, In the DPA V2 task, the default download directory is working directory and by design, it won't get cleaned up when running a new build.

[jahsu-MSFT]

@GarethOates I don't think I fully understand the scenario you described. Can you please provide a sample build definition/ yaml schema of the issue you ran into?

[GarethOates]

Sorry, I should have been a bit clearer about what my particular scenario is with regards to the 'build' and 'deploy' steps.

name: $(BuildDefinitionName).$(Build.SourceBranchName)$(Rev:.r)
trigger:
  batch: true
  branches:
    include:
    - master

stages:
- stage: BUILD
  jobs:
  - job: GetFiles
    displayName: Fetch the Template Files
    pool:
      vmImage: 'ubuntu-16.04'
    steps:
    - publish: $(System.DefaultWorkingDirectory)
      artifact: documents

- stage: TEST
  displayName: Deploy to TEST
  variables:
    SiteUrl: 'http://oursharepointarea.com/dokumenter'
    InnholdLibrary: 'DokumentMaler'
    FMLibrary: 'DokumentmalerFM'
  jobs:
  - deployment: TEST
    pool: TEST-Vera
    environment: TEST - SharePoint
    strategy:
      runOnce:
        deploy:
          steps:
            - download: current
              artifact: documents
            
            - task: PowerShell@2
              displayName: 'Upload Innhold Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(InnholdLibrary) $(Pipeline.Workspace)/documents/Innhold'
                failOnStderr: true

            - task: PowerShell@2
              displayName: 'Upload DocumentMalerFM Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(FMLibrary) ''$(Pipeline.Workspace)/documents/FM Templates'''
                failOnStderr: true

- stage: QA
  displayName: Deploy to QA
  variables:
    SiteUrl: 'http://oursharepointarea.com/dokumenter'
    InnholdLibrary: 'DokumentMaler'
    FMLibrary: 'DokumentmalerFM'
  jobs:
  - deployment: QA
    pool: QA-Vera
    environment: QA - SharePoint
    strategy:
      runOnce:
        deploy:
          steps:
            - download: current
              artifact: documents
            - task: PowerShell@2
              displayName: 'Upload Innhold Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(InnholdLibrary) $(Pipeline.Workspace)/documents/Innhold'
                failOnStderr: true

            - task: PowerShell@2
              displayName: 'Upload DocumentMalerFM Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(FMLibrary) ''$(Pipeline.Workspace)/documents/FM Templates'''
                failOnStderr: true

- stage: PROD
  displayName: Deploy to PROD
  variables:
    SiteUrl: 'http://oursharepointarea.com/dokumenter'
    InnholdLibrary: 'DokumentMaler'
    FMLibrary: 'DokumentmalerFM'
  jobs:
  - deployment: PROD
    pool: PROD-Vera
    environment: PROD - SharePoint
    strategy:
      runOnce:
        deploy:
          steps:
            - download: current
              artifact: documents
            
            - task: PowerShell@2
              displayName: 'Upload Innhold Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(InnholdLibrary) $(Pipeline.Workspace)/documents/Innhold'
                failOnStderr: true

            - task: PowerShell@2
              displayName: 'Upload DocumentMalerFM Documents'
              inputs:
                targetType: filePath
                filePath: '$(Pipeline.Workspace)/documents/Scripts/UploadDokument.ps1'
                arguments: '$(SiteUrl) $(FMLibrary) ''$(Pipeline.Workspace)/documents/FM Templates'''
                failOnStderr: true

I have a git repo which contains a collection of .docx and .pdf files. The yaml file I have here describes the process by which we upload those documents to our different SharePoint environments, using a simple Powershell script. This script loops over each .docx or .pdf file in the pipeline artifact directory and attempts to upload that file to SharePoint.

What I noticed when I updated the git repository to either include new files, or rename existing files, was that in deploy stages of my pipeline, the list of files being looped over and uploaded did not reflect the latest changes made to the repository, but instead was looping over an out of date collection of files.

I discovered that what was happening was, the 'download' task was not downloading the latest pipeline artifact produced in the previous stage, but was instead using a previously downloaded (and now old) cached version of the artifact.

So when my script was looping round all the files in the directory, those that were changed, added or renamed were not being included in this process.

I worked around this, by connecting to each agent and deleting the 'documents' artifact from the working directory. This then forced the download step to fetch the latest version of the documents artifact, which contained the correct files.

[jahsu-MSFT]

@GarethOates Thanks for the additional info. Can you please try adding dependsOn input field in the yaml schema and assign the stage name it's dependent on to the value and see if it works?
For example:

dependsOn: 
   - Build
   - Test
   - QA

For reference: https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema

[GarethOates]

I'm afraid that didn't fix the problem.

From the task output in Azure DevOps:
Download from the specified build: #1721
Download artifact to: C:_work\6/documents

But when I remote onto the on-premise agent's server and look at C:_work\6\documents, the Date modified of the 'documents' directory is 06.12.2019, the last time the build was run, not today's date. So it's using the old 'documents' artifact from the previous build, not the latest one published by the BUILD stage of my pipeline.

if I delete that directory, it will then fetch the latest one from the newest most recent build, and the task will succeed.

[jahsu-MSFT]

@GarethOates Thanks for the updates. I think that goes back to the scenario where old artifacts are not being deleted in the agent. In the DPA V2 task, the default download directory is working directory and by design, it won't get cleaned up when running a new build
I think you can try specifying $(System.ArtifactsDirectory) as the download directory, files inside of that folder should get cleaned up when the agent is running a new build/pipeline.

In the meantime, we will evaluate to see if it makes sense to set the default download directory to some folder that gets cleaned at the beginning of each run.

[GarethOates]

Ok, so does that mean I can’t use the shortcut version of those tasks like - download: current and must instead define the tasks fully, specifying the directory myself?

If so, I’ll do that. It might make sense to mention this caveat (if you can call it that) in the docs somewhere. I’ll have a go at updating them and submitting a PR tomorrow CET, unless you beat me to it, or don’t agree that it needs mentioning. 😊

[jahsu-MSFT]

@GarethOates Sorry for the inconvenience. Fro the download keyword, unfortunately the target input field is currently not exposed but it should have the capability for users to specify a custom download directory very soon.
For now if you'd like to use a different download directory, yes, you will need to define the task fully.

[jahsu-MSFT]

The issue should be resolved after adding dependent stages and regarding the default download directory not being cleaned up, it's currently by design. I'm closing the ticket for now and we can reopen it if you are still seeing the issue.

[GarethOates]

It was resolved by switching to using the DownloadPipelineArtifact task instead of the download keyword. Setting dependent stages did not resolve the issue, however I have been able to work around it.

[jahsu-MSFT]

@fadnavistanmay who will be looking into PA related issues. If this is download keyword related, the RM team can help look into it.

[trajano]

Referenced in https://stackoverflow.com/questions/62042095/how-do-you-download-all-pipeline-resources-in-a-job/62056385#62056385

[cawoodm]

If the solution is using the "DownloadPipelineArtifact task instead of the download keyword" then @microsoft you need to update your docs - this is driving folks crazy.