Azure Pipelines build system not recognized with non-Windows jobs

[gitfool]

See the logs for all jobs in Cake.Dungeon/#20190107.3:

• Windows
• Linux
• Docker
• Mac ❌

There are a couple of problems; one is that TF_BUILD is expected in TFBuildProvider.cs#L46 but is missing in the Docker job environment variables, and the other is that AGENT_NAME is always defined but not as expected in TFBuildProvider.cs#L82.

(The Mac job is currently broken for other reasons I haven't looked into yet.)

[devlead]

Wonder, is that an error in Azure Pipelines or per design?

Checking docs at https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts

Variable	Description
TF_BUILD	Set to True if the script is being run by a build task. This variable is agent-scoped. It can be used as an environment variable in a script and as a parameter in a build task, but not as part of the build number or as a version control tag.

Normally it's not available on the system but set by the process invoker on each task
https://github.com/Microsoft/azure-pipelines-agent/blob/148dbe4bb953da97a423ad899626c773371b3083/src/Agent.Sdk/ProcessInvoker.cs#L195

            // Set the TF_BUILD env variable.
            _proc.StartInfo.Environment["TF_BUILD"] = "True";

Looking at the docker bits it looks like they're just copying the host variables but not setting the extra "TF_BUILD=True"
https://github.com/Microsoft/azure-pipelines-agent/blob/30d2e8b9c2afe9cdb98d16016968616780cea933/src/Agent.Worker/Container/DockerCommandManager.cs#L123

So likely docker process is getting the variable, but not the container itself.

@gitfool maybe worth raising an issue on agent repo to ask.

[gitfool]

@devlead Done and already confirmed as a bug on their side. Re the second problem, since the AGENT_NAME is no longer simply Hosted Agent, I suggest changing the relevant part of TFBuildProvider.IsHostedAgent from:

_environment.GetEnvironmentVariable("AGENT_NAME") == "Hosted Agent"

to:

_environment.GetEnvironmentVariable("AGENT_NAME").StartsWith("Hosted")

Either that or switching to or also checking for the SYSTEM_SERVERTYPE environment variable, which I noticed has the value Hosted for all Azure Pipelines builds. The first option would be backwards compatible so would be preferred?

[devlead]

Isn't this from your build log?

2019-01-07T07:33:02.2586978Z AGENT_NAME                                                          Hosted Agent

Looks like Hosted Agent?

[gitfool]

Yes, for the Docker job, but the other logs use different hosts; currently the Linux host is Hosted Ubuntu 1604 2, and I've seen other variants, so the Linux job is detected as running on TFS, rather than VSTS / Azure Pipelines.

[gitfool]

I just noticed another problem. The TFBuildAgentInfo.MachineName is not set - a single " is how I log and distinguish empty strings - as it expects the AGENT_MACHINE_NAME environment variable, but it seems to be called AGENT_MACHINENAME instead.

[devlead]

Yes, for the Docker job, but the other logs use different hosts; currently the Linux host is Hosted Ubuntu 1604 2

I see, well yes then startswith makes sense probably _environment.GetEnvironmentVariable("AGENT_NAME")?.StartsWith("Hosted") as it will be null if the environment variable is not found.

I just noticed another problem. The TFBuildAgentInfo.MachineName is not set - a single " is how I log and distinguish empty strings - as it expects the AGENT_MACHINE_NAME environment variable, but it seems to be called AGENT_MACHINENAME instead.

AGENT_MACHINENAME seems to be correct spelling according to docs.

Variable	Description
Agent.MachineName	The name of the machine on which the agent is installed

https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts

[gitfool]

Okay, I'll submit a PR with those two changes...

[devlead]

@gitfool the mac build you can likely fix by adding a .NET installer task into yaml, example of such yaml

variables:
  DOTNET_SDK_VERSION: '2.1.502'

jobs:
- job: MacOS
  displayName: Build on MacOS

  pool:
    vmImage: 'macOS 10.13'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'

  - bash: ./build.sh
    displayName: 'Execute Cake Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'   
    inputs:
      testRunner: xUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true



- job: Windows
  displayName: Build on Windows

  pool:
    vmImage: 'VS2017-Win2016'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'


  - powershell: ./build.ps1
    displayName: 'Execute Cake PowerShell Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testRunner: XUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true


- job: Linux
  displayName: Build on Linux

  pool:
    vmImage: 'Ubuntu 16.04'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'


  - bash: ./build.sh
    displayName: 'Execute Cake Bash Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testRunner: XUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true


- job: Docker
  displayName: Build Docker Image

  pool:
    vmImage: 'Ubuntu 16.04'

  steps:
  - task: Docker@1
    displayName: 'Build image'

[gitfool]

@devlead thanks for the tip; it works on Mac again!

FTR, it only started failing when I switched to GitVersion.Tool. From the logs the Mac agent had up to .NET Core 2.2.0 installed. I always update to the latest tools, so I added a DotNetCoreInstaller task to install .NET Core SDK 2.2.102 and the issue went away. 😈