AzureFunctionApp@1 deployment gets stuck and fails after several retries

[ghost]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: AzureFunctionApp@1

Environment

• Azure Pipelines

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
    • msasg
    • Bing_UX
    • FastPath-Build-Test-Deploy-ALL
    • https://dev.azure.com/msasg/Bing_UX/_build/results?buildId=17945565

• Agent - Hosted or Private: Hosted

  • If using Hosted agent, provide agent queue name: vmImageName: 'windows-latest'

Issue Description

We are an internal Microsoft team and we are using the AzureFunctionApp@1 task to deploy our Function Apps in Azure. The issue consists in that many deployments of the App fail due to timeouts and failed retries.

It seems that the request to update the Function App package gets stuck and timeouts after around 3 mins (I guess this timeout is configured inside the task code) and after each timeout retries the request. This is retried several times and at the end, the task fails after 15-16 mins.

What we see in the task log is this:

This produces two issues:

• First, the deployment fails.
• Second, the Function App is left in a corrupted state where the new package has not been fully deployed, and the previous deployment is also broken, which results in the Function App not working anymore and required manual intervention.

We would like to know the root cause of this issue and also if it is possible to have some control from the AzureFunctionApp@1 input parameters to determine the duration of the timeouts and also the number of retries.

Also, the logs don't provide much detail on what the actual problem is.

Error logs

2021-01-07T23:53:09.2386187Z ##[section]Starting: Deploy Function App to PROD
2021-01-07T23:53:09.2639113Z ==============================================================================
2021-01-07T23:53:09.2639707Z Task : Azure Functions
2021-01-07T23:53:09.2640303Z Description : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2021-01-07T23:53:09.2640877Z Version : 1.163.7
2021-01-07T23:53:09.2641270Z Author : Microsoft Corporation
2021-01-07T23:53:09.2641786Z Help : https://aka.ms/azurefunctiontroubleshooting
2021-01-07T23:53:09.2642356Z ==============================================================================
2021-01-07T23:53:13.6702075Z Got service connection details for Azure App Service:'FastPathBasketballUx-prod4-east-us'
2021-01-07T23:53:15.0324558Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2021-01-07T23:53:15.0326563Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2021-01-07T23:53:15.3156558Z App Service Application settings are already present.
2021-01-07T23:53:21.9654603Z Package deployment using ZIP Deploy initiated.
2021-01-07T23:59:48.4868777Z ##[error]Failed to deploy web package to App Service.
2021-01-07T23:59:48.4977429Z ##[error]To debug further please check Kudu stack trace URL : https://$FastPathBasketballUx-prod4-east-us:***@fastpathbasketballux-prod4-east-us.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
2021-01-07T23:59:48.4981635Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)
2021-01-07T23:59:49.3382971Z App Service Application URL: http://fastpathbasketballux-prod4-east-us.azurewebsites.net
2021-01-07T23:59:49.3797056Z ##[section]Finishing: Deploy Function App to PROD

[zmarty]

Please be aware there is an internal e-mail thread discussion that contains more context, as well as links to specific code snippets that we think have issues.

[chshrikh]

@AmrutaKawade to check this on priority

[AmrutaKawade]

this is fixed. closing this issue

[sijucm]

This is not fixed and I'm suffering this Microsoftiness for the past 1.5 years. Now my application is in production and it embarasses me that this fails very randomly and I cannot even restart the deploy because of the controls put in place. It can take upto an hour for it to fail and I have to again start another deploy task and pray hard even being an atheist. The only wish I have is never to work with what comes out of Microsoft

[sijucm]

Microsoft support person tells this is fixed without even telling what is fixed and what was the root cause. From my experience with Microsoft support it is diversionary tactics to frustrate a customer to leave the issue.
Please explain how this is fixed. I can give you a 100 instance of this failing every day at least three times.

[justwiebe]

This is still failing for me too

[dikokob]

Same here

[asos-anton-hryschanka]

failing with the same results

[jfcroteau]

still failing with the same result. I saw the problem both on version 1.195.0 and 1.198.1.

[wkoeter]

My function deployments also fail sometimes with this exact result.

[runtimeware]

Same with us - although we typically don't wait very long and just cancel the job. Re-running will sometimes take care of it; however, unsure why the deployment stalls at that step.

[davemasters1984]

Currently experiencing this hang on 1.208.2

[pauldotknopf]

This is failing for me too, version 1.209.0, it worked previously on 1.208.2.

[yewtechnologiesuk]

I'm having same issue.

[ashanderson-sh]

I am experiencing the same thing for a second time:

2023-04-12T00:53:29.1565625Z ==============================================================================
2023-04-12T00:53:29.1566001Z Task         : Azure Functions
2023-04-12T00:53:29.1566112Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2023-04-12T00:53:29.1566334Z Version      : 1.216.1
2023-04-12T00:53:29.1566432Z Author       : Microsoft Corporation
2023-04-12T00:53:29.1566583Z Help         : https://aka.ms/azurefunctiontroubleshooting
2023-04-12T00:53:29.1566735Z ==============================================================================
2023-04-12T00:53:29.6603384Z Got service connection details for Azure App Service:'xxxxxxxxxxxxx'
2023-04-12T00:53:30.5729419Z (node:67971) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2023-04-12T00:53:33.6708146Z NOTE: Function app is VNet integrated.
2023-04-12T00:54:34.1859794Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2023-04-12T00:54:34.1860334Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2023-04-12T00:54:34.1860897Z App Service Application settings are already present.
2023-04-12T01:05:47.2415821Z Validating deployment package for functions app before Zip Deploy
2023-04-12T01:32:24.8892843Z Package deployment using ZIP Deploy initiated.
2023-04-12T01:53:22.1678990Z ##[error]The operation was canceled.
2023-04-12T01:53:22.1684620Z ##[section]Finishing: AzureFunctionApp
2023-04-12T01:53:23.1030206Z ##[section]Starting: Checkout xxx/xxx@main to s

This was working fine until yesterday and then, all of sudden, stopped working. The task just times out after an hour. In the meantime, I have a second pipeline that deploys the exact same functions to another Function App that is working.

Last time I experienced this I had to delete everything - Function App, Storage Account etc. - to get it back into a working state. So far this time, I have tried deleting the Function App and its file shares but with no success.

[PeterAndrewGilbert]

I am experiencing the same issue. Out of the blue, as of April 12th, my functionapp deployment pipelines all started timing out.