Azure Web Application not updated, action still claims "Success" #404

lassevk · 2024-03-21T10:22:17Z

We discovered that our changes were not appearing in our website, an Umbraco 13 website hosted in Azure earlier this week.

To deploy, we used the wizard in Azure to create the yml files in Github to to the build and deployment, and have since changed these slightly to get release build numbers. The actual scripts being run are not touched by us.

This week we experienced that after publishing a release, and waiting for the build+deploy action to complete, our changes were not deployed. The action claimed it was a success, but clearly it wasn't.

After digging, we discovered that kudusync had failed due to a locked file. Just to be clear, this issue is not about the locked file, that was just the underlying cause.

In the logs from the script execution on the host, I can see that:

Error: Failed to change file that is currently being used   
"C:\home\site\wwwroot\umbraco\Logs\UmbracoTraceLog.DW0MDWK0000DQ.20240318.json"  
Failed exitCode=1, command="kudusync" -v 50 -x -f "C:\local\Temp\zipdeploy\extracted" -t "C:\home\site\wwwroot" -n   
"C:\local\Temp\tmp73B5.tmp" -p "C:\local\Temp\tmp73B5.tmp" -i ".git;.hg;.deployment;deploy.cmd"  
An error has occurred during web site deployment.

The last text there, "An error has occurred during web site deployment" comes from the script, which thus seems to have correctly identified that kudusync failed, as it took a branch to display this error based on the errorlevel.

However, the githut action claims that deployment was successful:

Run azure/webapps-deploy@v3
Package deployment using OneDeploy initiated.
{
  id: 'xxxxxxxxxxxxxxxxxxxxxxxx',
  status: 4,
  status_text: '',
  author_email: 'N/A',
  author: 'N/A',
  deployer: 'OneDeploy',
  message: 'OneDeploy',
  progress: '',
  received_time: '2024-03-18T13:54:57.8413124Z',
  start_time: '2024-03-18T13:54:58.060074Z',
  end_time: '2024-03-18T13:55:14.6408492Z',
  last_success_end_time: '2024-03-18T13:55:14.6408492Z',
  complete: true,
  active: true,
  is_temp: false,
  is_readonly: true,
  url: 'https://??.scm.azurewebsites.net/api/deployments/latest',
  log_url: 'https://??.scm.azurewebsites.net/api/deployments/latest/log',
  site_name: '??',
  provisioningState: 'Succeeded'
}
Deploy logs can be viewed at https://??.scm.azurewebsites.net/api/deployments/xxxx/log
Successfully deployed web package to App Service.
App Service Application URL: https://??.azurewebsites.net

This lead to us spending wasteful time trying to identify the issue with our code, trying to see if we messed up testing, or we managed to do merging wrong, or whatnot, instead of looking at the actual problem.

So, exactly how to reproduce, I would imagine that if you were to be able to simulate that kudusync fails, you would have the same issue, as a number of retries we did all have this same error situation, and the same outcome.

We fixed it by shutting down the site completely during deployment, but it would be nice if the github action actually reported deployment failure, when deployment fails.

Attachments (let me know if you want me to dig up more details from whatnot and I'll provide everything I can)

github action output.txt

deployment script log.txt

deploy-logs.json

The text was updated successfully, but these errors were encountered:

lassevk · 2024-03-21T10:24:09Z

Seems the yml file didn't attach, renamed it to txt and attached it here

azure_lkweb-yml.txt

CarlSargunar · 2024-03-22T13:26:57Z

I had this exact same problem @lassevk

As a quick fix, if you revert to the V2 action, the deployment works successfully

This is the deploy step from our action

  # Deploy to UAT
  - name: Deploy to Azure WebApp
    uses: azure/webapps-deploy@v2
    with:
      app-name: ${{ env.AZURE_WEBAPP_NAME }}
      slot-name: 'prelive'
      publish-profile: ${{ secrets.AZURE_UAT_XXXXXX_PROFILE }}
      package: ${{ github.workspace }}\publish.zip

CarlSargunar · 2024-03-22T13:28:56Z

We noticed the problem (and it may be an umbraco specific one, or may be one more related to locked files on our side (particularly log files and nucache files)

lassevk · 2024-03-25T14:05:44Z

We noticed the problem (and it may be an umbraco specific one, or may be one more related to locked files on our side (particularly log files and nucache files)

The locked file is likely an Umbraco specific problem, or at least related to Umbraco, but the overall issue of a failed deployment being shown as successful is troublesome. If we can't trust the deployment action to actually tell us that there is an issue, we're forced to monitoring it every deployment. We'd rather just have to go look at it if it fails.

lassevk · 2024-03-25T14:23:46Z

I had this exact same problem @lassevk

As a quick fix, if you revert to the V2 action, the deployment works successfully

This is the deploy step from our action
  # Deploy to UAT
  - name: Deploy to Azure WebApp
    uses: azure/webapps-deploy@v2
    with:
      app-name: ${{ env.AZURE_WEBAPP_NAME }}
      slot-name: 'prelive'
      publish-profile: ${{ secrets.AZURE_UAT_XXXXXX_PROFILE }}
      package: ${{ github.workspace }}\publish.zip

I believe we upgraded because we were getting warnings about v2 being deprecated, I will need to confer with my colleague tomorrow if it was this action that we upgraded due to warnings or something else. Maybe I'm mixing actions here.

CarlSargunar · 2024-03-25T15:46:59Z

the overall issue of a failed deployment being shown as successful is troublesome

This is the same behaviour we were seeing - the action was reporting as successfully deploying, but the logs showed that it had failed because a locked file. Agreed that this is incorrect behaviour

CarlSargunar · 2024-03-25T15:50:09Z

I believe we upgraded because we were getting warnings about v2 being deprecated

I'd be curious about that - I've not seen or been able to find any detail - perhaps a mod or admin can help?

I also see that v3.0 back in October was released but nothing has been updated since : https://github.com/Azure/webapps-deploy/releases

If the webapp publish profile secret is empty, webapps-deploy v3 doesn’t deploy but reports the deploy as passing. See Azure/webapps-deploy [Issue #404](Azure/webapps-deploy#404). Configs were set before running the deploy. This means that the config values in Azure are updated even if the deploy fails. Also, as written, the action was runnable by anyone with write access. That is too broad for production. To avoid these known issues: * check that all required secrets are set before proceeding * only update configs if the deploy passes * call the reusable workflow that checks if the user has access to deploy

Several issues were identified with the workflow: * If the webapp publish profile secret is empty, webapps-deploy v3 doesn’t deploy but reports the deploy as passing. See Azure/webapps-deploy [Issue #404](Azure/webapps-deploy#404). * Configs were set before running the deploy. This means that the config values in Azure are updated even if the deploy fails. * As written, the action was runnable by anyone with write access. That is too broad for production. To avoid these known issues: * check that all required secrets are set before proceeding * call the reusable workflow that checks if the user has access to deploy _NOTE: Configs cannot be moved after the deploy because they are used by the restart process kicked of by the deploy._

Posiden104 · 2024-04-24T14:17:27Z

+1 on this. Extremely frustrating trying to figure out why the action reports success, but my changes aren't being applied. It appeared to be the same issue as lassevk, it was trying to modify a log file but couldn't.

I can confirm using v2 did solve the issue for me. It looks to me that v2 doesn't attempt to access or delete the log files

CarlSargunar · 2024-04-24T22:02:42Z

My concern is that anyone new to this is automatically going to go for the newest version, but that just doesn't work - the bug is repeatable and consistantly failing.

lassevk added the need-to-triage label Mar 21, 2024

This was referenced Apr 10, 2024

fix: deploy falsely reports SUCCESS when it failed clearlydefined/service#1091

Closed

add reusable workflow for dev deploy clearlydefined/operations#68

Closed

thekaveman mentioned this issue Aug 7, 2024

chore(deps): bump azure/webapps-deploy from 2 to 3 cal-itp/benefits#2114

Draft

stevematney mentioned this issue Sep 27, 2024

Github Deployment Marked as Successful, But Kudu Logs Show Failure microsoft/pipelines-appservice-lib#106

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Web Application not updated, action still claims "Success" #404

Azure Web Application not updated, action still claims "Success" #404

lassevk commented Mar 21, 2024 •

edited

Loading

lassevk commented Mar 21, 2024

CarlSargunar commented Mar 22, 2024 •

edited

Loading

CarlSargunar commented Mar 22, 2024

lassevk commented Mar 25, 2024

lassevk commented Mar 25, 2024

CarlSargunar commented Mar 25, 2024

CarlSargunar commented Mar 25, 2024

Posiden104 commented Apr 24, 2024

CarlSargunar commented Apr 24, 2024

Azure Web Application not updated, action still claims "Success" #404

Azure Web Application not updated, action still claims "Success" #404

Comments

lassevk commented Mar 21, 2024 • edited Loading

lassevk commented Mar 21, 2024

CarlSargunar commented Mar 22, 2024 • edited Loading

CarlSargunar commented Mar 22, 2024

lassevk commented Mar 25, 2024

lassevk commented Mar 25, 2024

CarlSargunar commented Mar 25, 2024

CarlSargunar commented Mar 25, 2024

Posiden104 commented Apr 24, 2024

CarlSargunar commented Apr 24, 2024

lassevk commented Mar 21, 2024 •

edited

Loading

CarlSargunar commented Mar 22, 2024 •

edited

Loading