Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure/arm-deploy@v1 fails after new release of Azure CLI Version 2.24.1 #48

Closed
JaMatus opened this issue Jun 2, 2021 · 59 comments
Closed
Assignees
Labels
default stale 90 days old

Comments

@JaMatus
Copy link

JaMatus commented Jun 2, 2021

Hello 🙂 Since the new release (1. June 2021) of Azure CLI Version 2.24.1, our github actions deployment to azure fails on azure/arm-deploy@v1 with following error:

The process 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd' failed because one or more lines were written to the STDERR stream

Have you experienced similar problem or this is not related in your opinion?

Thanks

@github-actions
Copy link

github-actions bot commented Jun 2, 2021

This issue is marked default for generating issues report.

@mburuckercSO
Copy link

We are running into a similar issue on Linux runners.
The process '/usr/bin/az' failed because one or more lines were written to the STDERR stream

@t-dedah
Copy link
Contributor

t-dedah commented Jun 2, 2021

@mburuckercSO @JaMatus Can you please share the workflow file?

@github-actions github-actions bot removed the default label Jun 2, 2021
@JaMatus
Copy link
Author

JaMatus commented Jun 2, 2021

@t-dedah sure 🙂 here is the release workflow file:

name: DEV .NET Core CD ARM template

on:
  push:
    branches: [ dev ]

jobs:
  build:
    runs-on: windows-latest

    steps:
    - uses: actions/checkout@v2

    - name: Set up .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1

    - name: Build with dotnet
      run: dotnet build --configuration Release

    - name: dotnet publish
      run: dotnet publish -c Release -o ${{env.DOTNET_ROOT}}/myapp

    - uses: actions/upload-artifact@v2
      with:
        name: my-artifact
        path: ${{env.DOTNET_ROOT}}/myapp

  deploy-dev:
    runs-on: windows-latest
    needs: build

    steps:
    - uses: actions/checkout@v2

    - uses: actions/download-artifact@v2
      with:
        name: my-artifact
        path: ${{env.DOTNET_ROOT}}/myapp

    - uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS_DEV }}

    - name: Run ARM deploy
      uses: azure/arm-deploy@v1
      id: arm-deploy
      with:
        subscriptionId: ${{ secrets.AZURE_SUBSCRIPTION }}
        resourceGroupName: ${{ secrets.AZURE_RG_DEV }}
        template: ./deployment/backend.json
        parameters: ./deployment/backend.dev.params.json
          sqlAdministratorLogin=${{ secrets.SQLADMINISTRATORLOGIN_DEV }}
          sqlAdministratorLoginPassword=${{ secrets.SQLADMINISTRATORLOGINPASSWORD_DEV }}
          jwtTokenSecret=${{ secrets.JWTTOKENSECRET_DEV }}
          sendgridEmailApiKey=${{ secrets.SENDGRIDEMAILAPIKEY_DEV }}

    - name: Get WebApp publishing profile
      run: |
        $publishingProfile = $( `
          az webapp deployment list-publishing-profiles `
            --name ${{ steps.arm-deploy.outputs.webAppName }} `
            --resource-group ${{ secrets.AZURE_RG_DEV }} `
            --subscription ${{ secrets.AZURE_SUBSCRIPTION }} `
            --xml `
        ) && `
        echo ::add-mask::$publishingProfile && `
        echo ::set-env name=PUBLISHING_PROFILE::$publishingProfile
    - name: Deploy to Azure Web App
      uses: azure/webapps-deploy@v2
      with:
        app-name: '${{ steps.arm-deploy.outputs.webAppName }}'
        slot-name: 'production'
        publish-profile: ${{ env.PUBLISHING_PROFILE }}
        package: ${{env.DOTNET_ROOT}}/myapp

@github-actions
Copy link

github-actions bot commented Jun 2, 2021

This issue is marked default for generating issues report.

@marvinbuss
Copy link

I am experiencing the same issue. It does not matter which ARM template you are deploying. It fails whatever you are trying to deploy.

@github-actions github-actions bot removed the default label Jun 3, 2021
@PaulCousinsTTEducation
Copy link

I am seeing this issue too. There seems to be a warning being displayed before the failure,not sure if it is related but here is the relevant log from a run today. I hadn't seen any warnings prior to todays runs.

2021-06-03T10:54:38.6823203Z ##[endgroup]
2021-06-03T10:54:40.5955013Z Validating template...
2021-06-03T10:54:45.7235801Z ##[warning]
2021-06-03T10:54:45.7983377Z Creating deployment...
2021-06-03T10:54:54.8856051Z ##[error]
2021-06-03T10:55:25.9608960Z ##[error]
2021-06-03T10:55:26.0173126Z ##[error]The process '/usr/bin/az' failed because one or more lines were written to the STDERR stream

@github-actions
Copy link

github-actions bot commented Jun 3, 2021

This issue is marked default for generating issues report.

@JCDeist
Copy link

JCDeist commented Jun 3, 2021

I am seeing this issue as well. I will mention the Azure portal is showing the deployment as succesful, and I have verified that the resources have been created/changed.

@marvinbuss
Copy link

I am seeing this issue too. There seems to be a warning being displayed before the failure,not sure if it is related but here is the relevant log from a run today. I hadn't seen any warnings prior to todays runs.

2021-06-03T10:54:38.6823203Z ##[endgroup]
2021-06-03T10:54:40.5955013Z Validating template...
2021-06-03T10:54:45.7235801Z ##[warning]
2021-06-03T10:54:45.7983377Z Creating deployment...
2021-06-03T10:54:54.8856051Z ##[error]
2021-06-03T10:55:25.9608960Z ##[error]
2021-06-03T10:55:26.0173126Z ##[error]The process '/usr/bin/az' failed because one or more lines were written to the STDERR stream

I can confirm this. Validation still works fine, but also shows the warning.

@elizabethlfransen
Copy link

I can confirm that I am seeing the same errors and my deployment was successful

@jlomen
Copy link

jlomen commented Jun 3, 2021

Seeing the same issue also this morning

@github-actions github-actions bot removed the default label Jun 3, 2021
@ashmind
Copy link

ashmind commented Jun 3, 2021

Seeing the same issue.
If it's related to az cli change, I think it would be good to add azcliversion parameter same as https://github.com/Azure/cli, so that az cli updates are not picked automatically.

@github-actions
Copy link

github-actions bot commented Jun 3, 2021

This issue is marked default for generating issues report.

@ashmind
Copy link

ashmind commented Jun 3, 2021

I wonder if it's related to Azure/azure-cli#18262 (though that was reported earlier, for 2.24.0).

@iamalexmang
Copy link

Seeing this issue as well. Is anyone aware of any workarounds?

@elizabethlfransen
Copy link

Seeing this issue as well. Is anyone aware of any workarounds?

If you don't need any outputs we simply used continue_on_error: true

@iamalexmang
Copy link

I did, @elizabethlfransen, but as it turns out the outputs are missing in the from the step.
Before you ask, I double checked that outputs actually exist from the deployment, and they do. However, they don't seem to be bound in the GH Action step.

@elizabethlfransen
Copy link

@iamalexmang Yeah when it errors out there wont be any outputs. I was lucky to be in a case where we didn't need the outputs

@github-actions github-actions bot removed the default label Jun 4, 2021
@github-actions
Copy link

github-actions bot commented Jun 4, 2021

This issue is marked default for generating issues report.

@parallo-mattallford
Copy link

Merged a temporary fix. Please use this until we get a permanent solution from azure-cli team.

- uses: azure/arm-deploy@main

instead of using v1

Hi @t-dedah, is there any ETA on a permanent fix for v1?

@github-actions github-actions bot removed the default label Jun 17, 2021
@github-actions
Copy link

This issue is marked default for generating issues report.

@t-dedah
Copy link
Contributor

t-dedah commented Jun 17, 2021

@parallo-mattallford @plzm @dico-harigkev fix has been released by azure-cli team but public agent pool is yet to be updated to use upgrade azure-cli, I am trying to get updates for this.

@t-dedah
Copy link
Contributor

t-dedah commented Jun 17, 2021

@dico-harigkev @plzm Can you please share more info to investigate better

  1. Which version of azure-cli is running on pool
  2. if possible, share the arm-template

@t-dedah
Copy link
Contributor

t-dedah commented Jun 17, 2021

This is the schedule for next image rollout for the public agents. After this new fix should be available to everyone. Please test with V1 after this.
Orgs with self-hosted agents should update their azure-cli to 2.25.0

Windows 2019 - Fri /Mon
Other platforms - Wed next week

@bishal-pdMSFT
Copy link
Contributor

@dico-harigkev @parallo-mattallford @plzm we are looking at the issue of action reporting success even if deployment fails.
In the meanwhile, a workaround @t-dedah mentioned above is to use a self hosted runner with installed CLI version 2.25.0

@alex-frankel
Copy link

alex-frankel commented Jun 17, 2021

@t-dedah / @bishal-pdMSFT - can we please look into setting the az cli version if the task is going to take that dependency? Otherwise, we expose our customers to breaking changes like this with no good workarounds.. Alternatively, can we take advantage of the version in the task name (i.e. azure/[email protected])? That is what it should be used for, no?

For the other folks on the thread that have experienced this issue, why not use the azure/CLI@v1 task and deploy the template directly with the az deployment * create ... command? This task gives you direct control over which CLI version you want to use. What is the benefit that the ARM deployment task is providing for you?

@Choc13
Copy link

Choc13 commented Jun 17, 2021

@t-dedah / @bishal-pdMSFT - can we please look into setting the az cli version if the task is going to take that dependency? Otherwise, we expose our customers to breaking changes like this with no good workarounds.. Alternatively, can we take advantage of the version in the task name azure/[email protected]? That is what it should be used for, no?

For the other folks on the thread that have experienced this issue, why not use the azure/CLI@v1 task and deploy the template directly with the az deployment * create ... command? This task gives you direct control over which CLI version you want to use. What is the benefit that the ARM deployment task is providing for you?

@ashmind and I made this point above about making the CLI version a parameter. I think it should be decoupled from the version number of this action though and supplied in the with: block.

I did briefly look into making this change, but it looks like the AZ cli is pre-installed on the hosts so this action is just implicitly relying on whatever version is already installed. I think that means it would be quite a bit of work to allow the version to be set as an install step for the CLI would have to be added first. Maybe a setup-az-cli action which people could run at the top of their workflow would be nice and it would just clobber any versions that are pre-installed on the machine.

I do agree with you though that this wrapper action seems to be causing more harm than good given it's such a thin layer of indirection.

@Choc13
Copy link

Choc13 commented Jun 17, 2021

Also I think all of the errors being reported here about the action reporting success even though the deployment failed are a direct consequence of the workaround that's been applied to main. I believe #50 disabled checking stderr because the CLI was erroneously writing whitespace to stderr, so it doesn't seem surprising that as a consequence of this workaround the action will now succeed even when they are errors.

@t-dedah
Copy link
Contributor

t-dedah commented Jun 17, 2021

@parallo-mattallford @plzm @dico-harigkev I tried to repro the issue with a template deployment but it succeeded for me. Will it be possible to provide me with repro steps ?

@iamalexmang
Copy link

iamalexmang commented Jun 17, 2021 via email

@plzm
Copy link

plzm commented Jun 17, 2021

Thanks @t-dedah - see a workflow run here:
https://github.com/plzm/always-on/runs/2850980721?check_suite_focus=true
Three steps failed ("Deploy NSG Flow Logs ***") but the run shows as successful.

I have run a variety of ARM templates in this repo, all exhibit the same behavior.

@github-actions github-actions bot removed the default label Jun 17, 2021
@t-dedah
Copy link
Contributor

t-dedah commented Jun 17, 2021

@parallo-mattallford @plzm @dico-harigkev We have fixed the main version of action to fail if arm deployment fails.

@github-actions
Copy link

This issue is marked default for generating issues report.

@t-dedah
Copy link
Contributor

t-dedah commented Jun 18, 2021

Fix for this issue has been released to v1 version.

@t-dedah
Copy link
Contributor

t-dedah commented Jun 18, 2021

Those who moved to main, please feel free to move back to v1 version.

- uses: azure/arm-deploy@v1

instead of using main

@github-actions
Copy link

This issue is stale because it has been open for 7 days with no activity.

@github-actions github-actions bot added the stale 90 days old label Jun 25, 2021
@t-dedah
Copy link
Contributor

t-dedah commented Jun 25, 2021

Closing this issue as fixed has been deployed.

@t-dedah t-dedah closed this as completed Jun 25, 2021
@bishal-pdMSFT
Copy link
Contributor

A RCA of this issue has been posted here. Please feel free to put your comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
default stale 90 days old
Projects
None yet
Development

No branches or pull requests