CopyPublishBuildArtifacts timed out after 00:05:00

[ToonDC]

I did a setup of an on-premises agent, but when I try to upload an Azure Cloud Service package with the task CopyPublishBuildArtifacts of approx 150MB on a less performant network than a hosted agent I get a timeout exception.

Following message is show in the build log view:

2016-01-21T11:14:12.7635628Z Start: UploadArtifact 
2016-01-21T11:14:12.7645230Z ##[error]The HTTP request timed out after 00:05:00. 
2016-01-21T11:14:12.7650233Z End: UploadArtifact

I looked in the build task, but cannot find a parameter for increasing the timeout, how can I solve this issue?

[chrispat]

At the moment there is no way to control the upload timeout from the task as the actual upload work is done by the agent. Thanks for reporting the issue, we will look into a fix.

From: Toon De Coninck [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 3:58 AM
To: Microsoft/vso-agent-tasks [email protected]
Subject: [vso-agent-tasks] CopyPublishBuildArtifacts timed out after 00:05:00 (#1129)

I did a setup of an on-premises agent, but when I try to upload an Azure Cloud Service package with the task CopyPublishBuildArtifacts of approx 150MB on a less performant network than a hosted agent I get a timeout exception.

Following message is show in the build log view:

2016-01-21T11:14:12.7635628Z Start: UploadArtifact

2016-01-21T11:14:12.7645230Z ##[error]The HTTP request timed out after 00:05:00.

2016-01-21T11:14:12.7650233Z End: UploadArtifact

I looked in the build task, but cannot find a parameter for increasing the timeout, how can I solve this issue?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1129.

[TingluoHuang]

@ToonDC I don't want to increase the default timeout, but I will try to make a change to let you config that http request timedout through environment variable on the build machine.
Just for confirm, you hit the timeout on Hosted agent or onprem agent?

[ToonDC]

@TingluoHuang onprem. The suggested solution would help us a lot.

[Jaans]

We're hitting this problem now for an On-Prem VSO build agent (TFS 2015 with VisualStudio.com backend for agent).

Is there any interim hack / workaround available as we have downstream release processes that depend on the build artefacts being present. This is a critical issue for us that is blocking releases?

[lsprado]

I found the same problema now, any workaround?

[ghost]

I know this isn't really adding anything to the discussion but is there so workaround for this available yet or is there one planned?

[bryanmacfarlane]

Assigned to @TingluoHuang for consideration in the new vsts-agent core clr command

[danbridgellc]

+1 for timeout issue, I am using OnPrem agent and a crappy DSL internet connection! Anxious for a workaround/fix

[Youngr6]

I'm hitting this too, only way Ive found to solve it is to reduce the size of the artifact as much as possible, in my case making publishing multiple artifacts for one build

[bryanmacfarlane]

Is this all windows agents? Just wanted to point out xplat agent has issues up until really recent 0.6.x version where it chunked uploads. Also windows agent will retry, xplat one won't. New combined one will do both. We will allow custom timeouts and share a private soon.

[Youngr6]

Yes Windows agent for me. I only installed it a couple of weeks ago. The problem is single large files like a web package, lots of smaller files seem ok.

[bryanmacfarlane]

Large files are broken into chunks with retries around each chunk. So large files are more susceptible simply because more time is spent making it more likely to be hit by a hiccup. We could also look at tuning chunk size, retry back offs etc

We should test with tools to inject network latency and hiccups

[TingluoHuang]

In the current agent code, we break all upload file into chuck, the chuck size is 16MB, and the 5 mins timeout is for each chuck, which means the agent is not able upload 16MB in 5 mins.
Can everyone hit this issue tells me what kind of agent are you using, your own agent or hosted agent?
Is the agent configured for an on-prem TFS or VSTS (visualstudio.com)?

[danbridgellc]

I am using VSTS and the xplat agent on OSX...I am using my own agent not hosted.

I am probably an extreme case because I live in a remote area and the best internet I can get is DSL with .75mb upload speeds! Although with a timeout of 5min per 16mb that really should be plenty - not sure that is how it is working based on the logs...how can I confirm?

[bryanmacfarlane]

If you are doing OS X make sure you have the very latest 0.6.x agent. It only started chunking very recently. The see the chunking in action you would have to use fiddler or Charles ( xplat triubleshooting section shows how). You would see a bunch of post/put with headers with byte ranges

[danbridgellc]

sorry I was not clear...I am using the .Net Core xplat agent preview 2.99.0 - based on what I read, I understood it had everything the old xplat agent but maybe it does not have this since it is a newer feature??

[bryanmacfarlane]

That one is based on the vsts http client code from win agent so it has chunked for awhile.

[danbridgellc]

interesting...looking at the logs again - the logs in VSTS indicate the HTTP 05:00 timeout error, however when I look at the agent logs on the Mac itself, I see this...(Inner Exception #1) System.AggregateException: Catch exception during upload log file or attachment. (The HTTP request timed out after 00:01:40.) ---> System.TimeoutException: The HTTP request timed out after 00:01:40.

I have attempted this build/publish many times now and looking at the agent logs I see it always fails with this message of timing out after exactly 1:40 each time - any ideas?

[TingluoHuang]

1:40 is the timeout for uploading log or attachment, it is not CopyPublishBuildArtifacts's timeout, do you have missing build log? or some log is missing chunks?

[bryanmacfarlane]

... And xplat / new agent logs are paged. We're mixing discussions and topics here

[danbridgellc]

@TingluoHuang - I am not aware of any missing logs, etc - if I remove the CopyPublishArtifact task the full build completes successfully.

@bryanmacfarlane - sorry if this going off topic - should I open a different issue to track this?

[bryanmacfarlane]

@danbridgellc - np. Let's take a step back. We will do some testing with flaky network injection.

In the meantime, if you have a network trace ( Charles on OSX , fiddler on win ) of a failed upload, email it to me. Bryanmac is alias at ms com. Well go over it

[danbridgellc]

@bryanmacfarlane - I was focused on getting my issue resolved so I have relocated my Mac to a site that does have very good internet and it is working just fine now! Sorry but I made the move prior to seeing your request for a network trace so I didn't capture that...it is about an hour away now so not sure when I will have time to get it for a trace capture.

The good news is that it all seems to be working fine on a fast internet connection!

[Youngr6]

@bryanmacfarlane - tried again with latest agent installed (windows) - same result -

7_Publish Artifact WebApp.zip

I'm guessing the file is just over 30mb - problem seems to be that although it's doing the upload ok (I can see network util going to full), the progress tracker's only looking at files that have completed rather than chunks of files, and when it sees it hasn't been able to do 1 file in 5 minutes it times out.

[Youngr6]

ps. I'm now having to workaround by using a VM in azure, since I cant do much more to reduce the size of the webdeploy.zip file

[Shagufta4786]

I hit the same issue a while back and the reason was the ISP. Certain ports are blocked on certain ISP. Issue was resolved by changing the ISP.

[deekob]

We have hit the same issue while developing Xamarin Forms IOS application with a local Mac Build agent. .....

[bryanmacfarlane]

In the new agent we've added retries and more diagnostic logging on failure. The next release 2.105.0 will have block (16MB) level retry as well

https://github.com/Microsoft/vsts-agent

[Youngr6]

Hi Brian, Mac reports agent.version 2.101.1 and windows at 1.104.1 - I think I recall the agent updates itself on windows, is the Mac one supposed to as well? Just wondering if I need to go re-download it/re-install it when 2.105.0 is released to the wild. (my workaround's fine for the windows builds but I'm stuck as far as iOS goes unless I start doing something like macincloud).

[deekob]

I Installed 2.104 on a Mac and had to do it from scratch as the previous version was not auto updating as per documentation. FYI also hanging out for 2.105 and although 2.104 had improved logging the same problem occured when uploading our Xamarin IOS ipa file into VSTS.

[bryanmacfarlane]

@deekob - the fix is is 2.105.1 which has been released for many days.

We changed it to retry on blocks (4 MB chunks) with 10 min T.O. and add a file level retry queue behind that.

[Youngr6]

@bryanmacfarlane tried it on the windows box and it worked like a charm, will let you know if there's a problem with the macOs one, otherwise thanks for resolving!

[deekob]

@bryanmacfarlane - All good, Thx.