SDL verification job failure

[JunTaoLuo]

• This issue is blocking
• This issue is causing unreasonable pain

We're seeing some failures during the extraction step of the SDL job:

##[error](NETCORE_ENGINEERING_TELEMETRY=Sdl) The 'Write-Host' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
##[error]PowerShell exited with code '1'.

For example: https://dev.azure.com/dnceng/internal/_build/results?buildId=590888&view=logs&j=7d9eef18-6720-5c1f-4d30-89d7b76728e9&t=4c19956c-5c86-5c24-85d7-25b7fad31d7f&l=13

[sunandabalu]

See something similar in arcade but with the build stage - https://dnceng.visualstudio.com/internal/_build/results?buildId=592226&view=results

[pranavkm]

Another similar one:

2020-04-09T10:42:15.6952731Z ##[error](NETCORE_ENGINEERING_TELEMETRY=Sdl) The term 'Write-PipelineTelemetryError' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
2020-04-09T10:42:21.4011317Z ##[error]PowerShell exited with code '1'.
2020-04-09T10:42:21.4562408Z ##[section]Finishing: Extract Blob Artifacts

https://dev.azure.com/dnceng/internal/_build/results?buildId=594421&view=logs&j=7d9eef18-6720-5c1f-4d30-89d7b76728e9&t=c49a855a-1eb9-5019-62bf-639a659fe361&l=15

[JohnTortugo]

Another similar one:

2020-04-09T10:42:15.6952731Z ##[error](NETCORE_ENGINEERING_TELEMETRY=Sdl) The term 'Write-PipelineTelemetryError' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
2020-04-09T10:42:21.4011317Z ##[error]PowerShell exited with code '1'.
2020-04-09T10:42:21.4562408Z ##[section]Finishing: Extract Blob Artifacts

https://dev.azure.com/dnceng/internal/_build/results?buildId=594421&view=logs&j=7d9eef18-6720-5c1f-4d30-89d7b76728e9&t=c49a855a-1eb9-5019-62bf-639a659fe361&l=15

This one is probably due to an import error in the sdl script: https://github.com/dotnet/arcade/blob/master/eng/common/sdl/extract-artifact-packages.ps1#L32

[JohnTortugo]

##[error](NETCORE_ENGINEERING_TELEMETRY=Sdl) The 'Write-Host' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
##[error]PowerShell exited with code '1'.

This one is weird. Due the error description:

... command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded.

I'm inclined to think this was a configuration problem in the build agent.

[sunandabalu]

@adiaaida @chcosta any ideas on the "## error NETCORE_ENGINEERING_TELEMETRY=Sdl" errors?

[chcosta]

Without digging in, I agree with @JohnTortugo assessment

[alexperovich]

The Write-Host command not found issue has only occured in one out of 594 builds for aspnetcore, and is very much a machine configuration issue. This build ran on an azure devops hosted pool so it isn't one of our build machines that we can try to get a repro.

The Write-PipelineTelemetryError issue is an import issue with the scripts. It appears to be caused by another error that was partially swallowed somehow. I can see the tail end of a stack trace. It has also only ocured in one build. We should probably add some more logging to this step so we can see what actually happened.

[alexperovich]

The Write-Host issue only happened once and is 100% caused by an issue on the AzDev hosted machines. If it happens again we can continue investigating.

The Write-PipelineTelemetryError issue is actually covering up a different problem, its obscuring the actual error. I opened #5254 to make the logging better in error cases. This error has also only happened once. If it comes back we can investigate, but it is a distinct issue from the Write-Host one.