[Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. #18859

AndrewBates666 · 2023-08-23T14:54:26Z

Task name

AzureFilecopy

Task version

5.0

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

Windows 2019 Datacenter

Operation system

Windows 2019 Datacenter

Question

Hi, we have started seeing a problem resolving the Sanitizer module in the above task. The AzureFileCopy task copies a file from a Storage Account onto the C:\temp folder on an Azure Windows VM running Server 2019 Datacenter.

The log implies that the module has been loaded:

2023-08-23T14:15:20.7037529Z ##[debug]Loading module from path 'D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.227.0\ps_modules\Sanitizer\Sanitizer.psm1'.
2023-08-23T14:15:20.7118086Z ##[debug]Loading resource strings from: D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.227.0\ps_modules\Sanitizer/module.json

But when the script is attempting to run on the VM the following error is reported:

2023-08-23T14:16:43.6557692Z -ExecutionPolicy Unrestricted -Command ". 
2023-08-23T14:16:43.6558018Z 'C:\Users\adminuser\AppData\Local\Temp\e9f131e0-0904-4611-b60a-30f412e6b24b.ps1'"
2023-08-23T14:16:43.6558122Z 
2023-08-23T14:16:43.7348294Z Import-Module : The specified module 'C:\Users\adminuser\AppData\Local\Temp\ps_modules\Sanitizer' 
2023-08-23T14:16:43.7348885Z At line:1 char:1
2023-08-23T14:16:43.7349175Z + & 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo ...
2023-08-23T14:16:43.7349583Z + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-08-23T14:16:43.7350327Z     + CategoryInfo          : NotSpecified: (Import-Module :...les\Sanitizer' :String) [], RemoteException
2023-08-23T14:16:43.7350704Z     + FullyQualifiedErrorId : NativeCommandError
2023-08-23T14:16:43.7350922Z  
2023-08-23T14:16:43.7351127Z 
2023-08-23T14:16:43.7455112Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:43.7455461Z 
2023-08-23T14:16:44.5304389Z was not loaded because no valid module file was found in any module directory.
2023-08-23T14:16:44.5304948Z 
2023-08-23T14:16:44.5307796Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5308138Z 
2023-08-23T14:16:44.5308764Z At C:\Users\adminuser\AppData\Local\Temp\4cb675ed-e14c-457a-afc7-0a219eb60a0b.ps1:11 char:5
2023-08-23T14:16:44.5309112Z 
2023-08-23T14:16:44.5311649Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5311802Z 
2023-08-23T14:16:44.5312085Z +     Import-Module $PSScriptRoot\ps_modules\Sanitizer
2023-08-23T14:16:44.5312166Z 
2023-08-23T14:16:44.5312943Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5313071Z 
2023-08-23T14:16:44.5313211Z +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-08-23T14:16:44.5313283Z 
2023-08-23T14:16:44.5313848Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5314121Z 
2023-08-23T14:16:44.5314302Z     + CategoryInfo          : ResourceUnavailable: (C:\Users\adminu...dules\Sanitizer:String) [Imp 
2023-08-23T14:16:44.5314404Z 
2023-08-23T14:16:44.5315085Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5315202Z 
2023-08-23T14:16:44.5315430Z    ort-Module], FileNotFoundException
2023-08-23T14:16:44.5315497Z 
2023-08-23T14:16:44.5315880Z ##[debug]Processed: ##vso[task.complete result=Failed]
2023-08-23T14:16:44.5316004Z 
2023-08-23T14:16:44.5316182Z     + FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

Any advice on how to resolve this problem? Thanks Andy

starkmsu · 2023-08-24T07:43:01Z

@AndrewBates666 , please post exact task version that gives you that error. I mean like this - 5.xxx.yyy.

AndrewBates666 · 2023-08-24T08:06:33Z

It is the standard Azure File Copy task in Azure release pipelines; the only version info I can see is 5.*. Where can I see the full version? Also is it possible to use an old version of a task. This seems to be a problem that is starting to affect other users. See here: #18809

v-mohithgc · 2023-08-24T08:12:09Z

It is the standard Azure File Copy task in Azure release pipelines; the only version info I can see is 5.*. Where can I see the full version? Also is it possible to use an old version of a task. This seems to be a problem that is starting to affect other users. See here: #18809

Hi, the task version will be visible in task logs, for example:

can you please check and confirm the task version in your recent failure pipelines?

Thanks

AndrewBates666 · 2023-08-24T08:13:59Z

5.227.0.

LeftTwixWand · 2023-08-24T09:29:34Z

@AndrewBates666 are you sure that you use Microsoft-hosted agent, not self-hosted agent?

marcusrichards · 2023-08-24T09:41:22Z

We are having the exact same problem on all our pipelines, which are running as Microsoft-hosted agents. Highly critical as it is blocking our VM deployments.

AndrewBates666 · 2023-08-24T09:45:19Z

We're using the standard Azure DevOps Microsoft-hosted agents and release pipeline. All tasks using AzureFilecopy are failing with the Sanitizer error. Yes, all of our VMs can't currently be deployed so it is critical for us as well.

v-mohithgc · 2023-08-24T10:30:45Z

Hello everyone, please check and try to run your pipelines now, and if your using Microsoft hosted agents, task should be able to fetch version 5.226.1 (previous working version)
If your using self-hosted agent, please try to clear the cache of 5.227.0 and agents should be able to pick task version 5.226.1.
If you're using YAML pipeline, you can define the task version as 5.226.1 and try to run it.

Thanks!

marcusrichards · 2023-08-24T10:44:51Z

How do we change the classic pipeline task version from 5.* to 5.226.1 ? The drop down for task version only has 5., 4., 3* etc?

v-mohithgc · 2023-08-24T10:51:21Z

How do we change the classic pipeline task version from 5.* to 5.226.1 ? The drop down for task version only has 5., 4., 3* etc?

there is no need to change, if your using Microsoft hosted agent you can just re-run the pipeline and the agent should be able to automatically pick version 5.226.1.
Once you re-run let me know if your agent is not able to pick task version 5.226.1.
you can verify the task version in task logs only.
for example:

marcusrichards · 2023-08-24T10:57:28Z

I reran and it picked up 5.226.1 but error : throw "Failed while downloading azcopy.exe from the URL w .

I will upload log file now

marcusrichards · 2023-08-24T11:04:17Z

171377_logs_14.txt

I have had to remove some security information so new upload

v-mohithgc · 2023-08-24T11:15:33Z

171377_logs_14.txt

I have had to remove some security information so new upload

@marcusrichards going by the error message, this doesn't look to be an issue from the task end, it's something to do with your pipeline configuration.

@AndrewBates666 and others can you all confirm if the reverted version of 5.226.1 is showing a similar error or working fine for you all?

PonYehudak · 2023-08-24T11:18:26Z

i have the same problem, but i am using task# 2, and it is still picking 2.227.3 and not 2.226.1
how can i get it to pick up 2.226.1?

v-mohithgc · 2023-08-24T11:20:03Z

i have the same problem, but i am using task# 2, and it is still picking 2.227.3 and not 2.226.1 how can i get it to pick up 2.226.1?

AzureFileCopyV2 hasn't been reverted to 226.1 yet, please try if V4 or V5 version can be used.

PonYehudak · 2023-08-24T11:23:14Z

@v-mohithgc
when i use v4 or v5 i get other errors, i don't have time to try and solve them now.
When will V2 be reverted?

Griffa87 · 2023-08-24T11:24:58Z

@v-mohithgc testing this now. Pipeline is running so will update once complete

v-mohithgc · 2023-08-24T11:33:40Z

when i use v4 or v5 i get other errors, i don't have time to try and solve them now. When will V2 be reverted?

I have informed the relevant team, they're on it, meanwhile can you confirm the version of your last success run using v2?

Griffa87 · 2023-08-24T11:38:08Z

I can confirm it now works @v-mohithgc using AzureFileCopy 5.226.1.

PonYehudak · 2023-08-24T11:50:18Z

when i use v4 or v5 i get other errors, i don't have time to try and solve them now. When will V2 be reverted?

I have informed the relevant team, they're on it, meanwhile can you confirm the version of your last success run using v2?

2.226.1

Griffa87 · 2023-08-24T11:54:01Z

I can confirm it now works @v-mohithgc using AzureFileCopy 5.226.1.

One thing we have noticed @v-mohithgc since using 5.226.1 are a few warnings. I am not sure if they are caused by us or the AzureFileCopy code. But they are as follows.

2023-08-24T11:35:45.3699679Z ##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide

2023-08-24T11:36:23.1642944Z ##[warning]The names of some imported commands from the module 'Microsoft.Azure.PowerShell.Cmdlets.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.

2023-08-24T11:36:26.4002205Z ##[warning]The names of some imported commands from the module 'Az.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.

LeftTwixWand · 2023-08-24T12:10:09Z

@AndrewBates666 and @PonYehudak can you also share the link of failing pipeline, so we can investigate it further.
The format should be like this:
https://dev.azure.com/{organization}/{project}/_build/results?buildId={id}&view=results

PonYehudak · 2023-08-24T12:15:30Z

@LeftTwixWand
Is this OK -
https://pointerdev.visualstudio.com/PWFL-Platforms/_releaseProgress?_a=release-environment-logs&releaseId=14415&environmentId=39139

Griffa87 · 2023-08-24T13:04:09Z

@LeftTwixWand Andrew and I are on the same team and its working our side. Thanks

marcusrichards · 2023-08-24T13:32:32Z

171377_logs_14.txt
I have had to remove some security information so new upload

@marcusrichards going by the error message, this doesn't look to be an issue from the task end, it's something to do with your pipeline configuration.

Thank you. I upgraded the remote VM and everything is now running ok. Thanks for your super quick response 👍

LeftTwixWand · 2023-08-24T14:29:22Z

Hey @Griffa87, @AndrewBates666 and @PonYehudak thank you all for the help.

I would like to ask you to share the full pipeline log of your first run, where you faced the issue.
You can download it here:

If you are concerned about safety, you can send it on my Microsoft email: [email protected]

Griffa87 · 2023-08-24T14:45:14Z

I have emailed you @LeftTwixWand

PonYehudak · 2023-08-24T14:47:14Z

@LeftTwixWand - Also I have emailed you

v-mohithgc · 2023-08-25T06:23:51Z

@v-mohithgc when i use v4 or v5 i get other errors, i don't have time to try and solve them now. When will V2 be reverted?

V2 has been reverted to 2.226.1, please rerun and check.

Thanks

LeftTwixWand · 2023-08-25T10:24:19Z

Hey @Griffa87 and @PonYehudak thank you so much for sharing these details.
It's extremely helpful.

AlehPatskevich · 2023-08-26T21:23:45Z

Try to add addition azure powershell task with inline sript before your azure file copy task
Azcopy login --service-principal --application-id "your subscription application id" --tenant-id="your tenant" --aad-endpoint "https://login.windows.net/"

Have another problem but found this bug.

PonYehudak · 2023-08-27T06:28:51Z

@v-mohithgc when i use v4 or v5 i get other errors, i don't have time to try and solve them now. When will V2 be reverted?

V2 has been reverted to 2.226.1, please rerun and check.

Thanks

Works now,
Thank you.

AndrewBates666 · 2023-09-06T11:45:40Z

AzureFileCopy has started breaking again today; converting useSanitizerActivate cannot convert System.String :( From: PonYehudak ***@***.***> Sent: Sunday, August 27, 2023 7:29 AM To: microsoft/azure-pipelines-tasks ***@***.***> Cc: Andrew Bates ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/azure-pipelines-tasks] [Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. (Issue #18859) CAUTION: External email. Please do not click on links or attachments unless you know the content is genuine and safe. @v-mohithgc<https://github.com/v-mohithgc> when i use v4 or v5 i get other errors, i don't have time to try and solve them now. When will V2 be reverted? V2 has been reverted to 2.226.1, please rerun and check. Thanks Works now, Thank you. - Reply to this email directly, view it on GitHub<#18859 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BCC662QVLOMNX4ENKOIT5NDXXLSK7ANCNFSM6AAAAAA33TP5AA>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> Emails and their contents may be monitored for the purposes of quality control, fact checking and training. This email, together with any attachments, is confidential and may be privileged. If you have received it in error, please notify the sender and delete it. Please click here<https://trl.co.uk/permanent-landing-pages/privacy-notice> to view the TRL Privacy Notice. TRL Limited, registered in England, No. 3142272, registered office: Crowthorne House, Nine Mile Ride, Wokingham, RG40 3GA, UK. VAT Registration 664 625 321.

v-mohithgc · 2023-09-20T09:05:12Z

This issue is now resolved.
V228.1 will be containing the fix, which will be out by tomorrow to all the regions.
Feel free to reopen this issue if needed.

github-actions bot added Area: Release triage labels Aug 23, 2023

v-mohithgc added Area:RM RM task team Task:AzureFileCopy p1 Priority 1 (should have immediate attention) and removed triage labels Aug 25, 2023

LeftTwixWand mentioned this issue Aug 28, 2023

Revert "Additional Import for Sanitizer" #18886

Merged

2 tasks

AndrewBates666 mentioned this issue Sep 6, 2023

[BUG]: AzureFileCopy is failing again copying files from storage to a VM. #18924

Closed

4 tasks

v-mohithgc closed this as completed Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. #18859

[Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. #18859

AndrewBates666 commented Aug 23, 2023

starkmsu commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023 •

edited

Loading

marcusrichards commented Aug 24, 2023

marcusrichards commented Aug 24, 2023 •

edited

Loading

v-mohithgc commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

PonYehudak commented Aug 24, 2023 •

edited

Loading

Griffa87 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023 •

edited

Loading

PonYehudak commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

v-mohithgc commented Aug 25, 2023 •

edited

Loading

LeftTwixWand commented Aug 25, 2023

AlehPatskevich commented Aug 26, 2023

PonYehudak commented Aug 27, 2023

AndrewBates666 commented Sep 6, 2023 via email

v-mohithgc commented Sep 20, 2023

[Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. #18859

[Question]: AzureFileCopy started to fail on 21st Aug @ just after 12:30 BST. #18859

Comments

AndrewBates666 commented Aug 23, 2023

Task name

Task version

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server Version (if applicable)

Operation system

Question

starkmsu commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

AndrewBates666 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023 • edited Loading

marcusrichards commented Aug 24, 2023

marcusrichards commented Aug 24, 2023 • edited Loading

v-mohithgc commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

PonYehudak commented Aug 24, 2023 • edited Loading

Griffa87 commented Aug 24, 2023

v-mohithgc commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023 • edited Loading

PonYehudak commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

marcusrichards commented Aug 24, 2023

LeftTwixWand commented Aug 24, 2023

Griffa87 commented Aug 24, 2023

PonYehudak commented Aug 24, 2023

v-mohithgc commented Aug 25, 2023 • edited Loading

LeftTwixWand commented Aug 25, 2023

AlehPatskevich commented Aug 26, 2023

PonYehudak commented Aug 27, 2023

AndrewBates666 commented Sep 6, 2023 via email

v-mohithgc commented Sep 20, 2023

v-mohithgc commented Aug 24, 2023 •

edited

Loading

marcusrichards commented Aug 24, 2023 •

edited

Loading

PonYehudak commented Aug 24, 2023 •

edited

Loading

LeftTwixWand commented Aug 24, 2023 •

edited

Loading

v-mohithgc commented Aug 25, 2023 •

edited

Loading