nx affected -- Azure Pipelines

[marc-wilson]

Documentation issue

• Reporting a typo
• Reporting a documentation bug
• Documentation improvement
• Documentation feedback

Is there a specific documentation page you are reporting?

Enter the URL or documentation section here.
https://nx.dev/recipes/ci/monorepo-ci-azure#configuring-ci-using-azure-pipelines-and-nx

Additional context or description

Provide any additional details here as needed.

The documentation currently has this snippet:

trigger:
  - main
pr:
  - main

variables:
  CI: 'true'
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(System.PullRequest.PullRequestId) # You can use $(System.PullRequest.PullRequestNumber if your pipeline is triggered by a PR from GitHub ONLY)
    TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')]
  ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(Build.SourceBranchName)
  HEAD_SHA: $(git rev-parse HEAD)

jobs:
  - job: main
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Set Azure Devops CLI default settings
      - bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
      displayName: 'Set default Azure DevOps organization and project'

      # Get last successfull commit from Azure Devops CLI
      - bash: |
        LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
        if [ -z "$LAST_SHA" ]
        then
          LAST_SHA=$DEFAULT_BASE_SHA
        fi
        echo "Last successful commit SHA: $LAST_SHA"
        echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
      displayName: 'Get last successful commit SHA'
      env:
        AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      - script: npm ci

      - script: npx nx format:check --base=$(BASE_SHA)

      - script: npx nx affected --base=$(BASE_SHA) -t lint --parallel=3
      - script: npx nx affected --base=$(BASE_SHA) -t test --parallel=3 --configuration=ci
      - script: npx nx affected --base=$(BASE_SHA) -t build --parallel=3

• What is TARGET_BRANCH for? There is no usage of it after it is assigned.
• What is HEAD_SHA for? There is no usage of it after it is assigned.
• What is $DEFAULT_BASE_SHA? It is used, but not defined, assigned, or explained.
• What is CI? There is no usage of it after it is assigned.

[kklocker]

I have also been scratching my head regarding this example pipeline. A bit frustrating that it doesn't work "out of the box"

I would guess that HEAD_SHA is supposed to be in a --head=$(HEAD_SHA) of all npx nx scripts?

However there are other issues here:
I tried adding DEFAULT_BASE_SHA: $(git rev-parse HEAD~1), this ends up as the string "git rev-parse HEAD~1" instead of the computed value in the pipeline

Pleease 🙏 It would be really nice if the official documentation would have a working example

[marc-wilson]

This command also fails to work:

LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo."ci.sourceSha"")

• --organization must be specified
• --project must be specified

After defining both, you are still met with the mysterious $(DEFAULT_BASE_SHA) (no result)

[jstnjs]

Did you manage to figure it out?

[lukket]

With the following steps, it seems to work. None of the variables is needed if you don't use Nx Cloud. Regarding the documentation of the affected command, base is usually your main branch and head is HEAD.

- checkout: self
  fetchDepth: 0
- template: 'azure-pipelines-node-setup.yml'
# Set Azure Devops CLI default settings
- bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
  displayName: 'Set default Azure DevOps organization and project'
# Get last successfull commit from Azure Devops CLI
- bash: |
    git branch --track main origin/main
    DEFAULT_BASE_SHA=main
    LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
    if [ -z "$LAST_SHA" ]
    then
      LAST_SHA=$DEFAULT_BASE_SHA
    fi
    echo "Last successful commit SHA: $LAST_SHA"
    echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
  displayName: 'Get last successful commit SHA'
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

[surajkrishan]

Hi nx team any update : (

[jstnjs]

In my case I just needed it for a pull request. I removed almost everything and just used Build.SourceBranchName and Build.TargetBranchName on the nx affected commands.

But I believe you can make it work with the SHA as well. The reason the example on nx.dev is not working, is because you can't use git commands into the azure variables directly, but you could set them as a variable in a step/script.

[marc-wilson]

I've been playing around with this quite a bit and in short, here's the minimum pipeline that works for me.

This assumes you want the build to trigger on any commits to master or any pr's into master.

It is still a mystery as to what $DEFAULT_BASE_SHA is but from what I've gathered, it's irrelevant.

trigger:
  - master
pr:
  - master

variables:
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(System.PullRequest.PullRequestId)
  ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(Build.SourceBranchName)

stages:
  - stage: Affected
    displayName: 'Build, Test, Lint and Format'
    jobs:
      - job: Affected
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: git branch --track master origin/master
            displayName: 'Track master'

          - bash: |
              LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
              if [ -z "$LAST_SHA" ]
              then
                LAST_SHA=$DEFAULT_BASE_SHA
              fi
              echo "Last successful commit SHA: $LAST_SHA"
              echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
            displayName: 'Get last successful commit SHA'
            env:
              AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

          - script: npm ci
            displayName: 'Install Dependencies'

          - script: npx nx format:check --base=$(BASE_SHA)
            displayName: 'Check formatting'

          - script: npx nx affected --base=$(BASE_SHA) -t lint --parallel=3
            displayName: 'Lint'

          - script: npx nx affected --base=$(BASE_SHA) -t test --parallel=3 --configuration=ci
            displayName: 'Test'

          - script: npx nx affected --base=$(BASE_SHA) -t build -c production --parallel=3
            displayName: 'Build'

In my opinion, the real magic happens after all of this when you have your build artifact (your dist). Figuring out how to get that into production is fairly confusing. In my case, I use docker and after nx affected runs, I still need to build and push docker containers. I have "child" pipelines that watch for specific build tags for that process.

One thing that is worth mentioning is this step:

LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")

If that command is failing to work for you, check the build logs. It was random, but in the beginning it forced me to specify some extra arguments such as --organization. So if that is failing, it may help to run that command locally to figure out what exactly is going on. But, ultimately, that should give you the last SHA that was built successfully on your target branch. By target branch, this means master if it's a commit into master or your PR branch if your in PR. And again, simply playing with the command locally will give you a better idea of what is happening.

[siMs0n]

We tried your command but triggerInfo was empty for us. This command worked instead:
LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].sourceVersion")

The build contained the field "sourceVersion" which contained the commit SHA

[Ryandev]

Just to add, I had issues getting the NX provided snippet to work. I end up with az pipelines returning an empty value

This works for me however

    - task: CmdLine@2
      displayName: 'Get last successful commit SHA (commit)'
      condition: ne(variables['Build.Reason'], 'PullRequest')
      env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
      inputs:
          script: |
              BASE_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
              [ ! -n "$BASE_SHA" ] && echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA" && BASE_SHA=$(git rev-parse HEAD~1)
              echo "Using commit SHA: $BASE_SHA"
              echo "##vso[task.setvariable variable=BASE_SHA]$BASE_SHA"

    # TARGET_BRANCH_NAME=get just the branch name i.e. refs/origin/develop => develop
    - task: CmdLine@2
      displayName: 'Get last successful commit SHA (PR)'
      condition: eq(variables['Build.Reason'], 'PullRequest')
      inputs:
          script: |
              export TARGET_BRANCH_NAME=$(echo "$(System.PullRequest.TargetBranch)" | rev | cut -d'/' -f1 | rev)
              export TARGET_BRANCH="origin/$TARGET_BRANCH_NAME"
              export BASE_SHA=$(git merge-base "$TARGET_BRANCH" HEAD)
              echo "##vso[task.setvariable variable=BASE_SHA]$BASE_SHA"
              echo "##vso[task.setvariable variable=HEAD_SHA]$HEAD_SHA"
              echo "##vso[task.setvariable variable=TARGET_BRANCH]$TARGET_BRANCH
....
# run stages referencing BASE_SHA now i.e. `npx nx affected --base=$(BASE_SHA) -t lint`

[isaacplmann]

The Azure CI docs have been updated a few times since this issue was filed (including one update in progress #23453). Please open a new issue if there are still problems with the current documentation.

[surajkrishan]

the docs is still wrong it does not works for the PR

[isaacplmann]

What is the error you get for a PR?

[surajkrishan]

What is the error you get for a PR?

there is no error but for PR affected build is building for 0 project although there are lot of changes happen to different apps.

i am using this cmd in pipeline

npx nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) -t build --parallel=3

[isaacplmann]

That would happen if the base and head are not being set correctly. Can you try using LAST_SHA instead of BASE_SHA? Like this:

npx nx affected --base=$(LAST_SHA) --head=$(HEAD_SHA) -t build --parallel=3

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.