Timeout settings IN paket.dependencies

[tebeco]

Hi there,

Description

In our team we got nightly teamcity step that :

• clone some repo on our corp github
• run a paket update on some nuget
• build+Test ( using Fake)
• If ok, Commit and push

Since v5 of paket and the introduction of TimeOut we're facing serious issue
The big one called Nexus, it is a mess here ... (slow or unresponsive randombly but frequntly)

Also our Build Agent are created on the fly on our cloud so Setting a ENV VARIABLE is kind of a big deal (we could add it to install script but it's a bit too magic) it's a bit overkill

is there a way i missed a command parameter for paket update or in the paket.dependencies to be able to tweak that ?

Also is there a way to know what actually timeout (nuget + source + version + group ?)

Repro steps

Create a paket.depencies with 20-50 nuget inside paket files
Then add sources like nuget.org and other <= we use between 1 to 10 sources
Add failing source (randomly but at lest one everytime) (lower the timout ?)
And find a way to make it fail on Paket Install / Update

Then try to see what happened for paket output ;D
(see bellow)

Actual behavior

Paket failed with:
-> Unable to retrieve package versions for 'SomeTeamNuget'
-> TimeoutException: Waited 30 seconds for a request to finish.
Check the following sources, they might be rate limiting and stopped responding:
- https://api.nuget.org/v3/index.json
- https://someNugetSource1
- https://someNugetSource2
- https://someNugetSource3
Running build failed.
Error:
System.Exception: 'paket update nuget SomeTeamNuget' failed
at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1360.Invoke(String message)
at (our build fake step)
at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc2 action, IEnumerable1 source)
at (our build fake step)
at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target)

[forki]

with latest paket we changed that to 180s

[tebeco]

I will not bad mouth about guys maintains our nexus
Also we're using paket magic mode so I guess agent always runs latest version I think (I hope)

[forki]

could you please check logs for paket version?

[tebeco]

Paket version 5.2.1

Full detail is :

[02:01:31][Step 2/2] SomeStep (3m:02s)
[02:01:31][SomeStep] Target: UpdateTeamDependencies
[02:01:31][SomeStep] Starting Target: UpdateTeamDependencies (==> Clone)
[02:01:31][SomeStep] paket.exe update nuget SomeNuget
[02:01:32][SomeStep] Paket version 5.2.1
[02:01:32][SomeStep] Updating SomeNuget in D:\BuildAgent\.......\paket.dependencies group Main
[02:01:32][SomeStep] Resolving packages for group Main:
[02:04:33][SomeStep] Performance:
[02:04:33][SomeStep]  - Resolver: 3 minutes, 1 second (1 runs)
[02:04:33][SomeStep]     - Runtime: 1 second
[02:04:33][SomeStep]     - Blocked (retrieving package versions): 3 minutes (1 times)
[02:04:33][SomeStep]  - Average Request Time: 41 seconds
[02:04:33][SomeStep]  - Number of Requests: 97
[02:04:33][SomeStep]  - Runtime: 3 minutes, 2 seconds
[02:04:33][SomeStep] Paket failed with:
[02:04:33][SomeStep] -> Unable to retrieve package versions for 'SomeNuget'
[02:04:33][SomeStep] -> TimeoutException: Waited 30 seconds for a request to finish.
[02:04:33][SomeStep]          Check the following sources, they might be rate limiting and stopped responding:
[02:04:33][SomeStep]           - https://api.nuget.org/v3/index.json
[02:04:33][SomeStep]           - https://HellIsHostedHere
[02:04:33][SomeStep]           - https://OrHere
[02:04:33][SomeStep]           - https://AndHere
[02:04:33][SomeStep] Running build failed.
[02:04:33][SomeStep] Error:
[02:04:33][SomeStep] System.Exception: 'paket update nuget SomeNuget' failed
[02:04:33][SomeStep]    at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1360.Invoke(String message)
[02:04:33][SomeStep]    at OurBuildFakeStackTrace
[02:04:33][SomeStep]    at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source)
[02:04:33][SomeStep]    at OurBuildFakeStackTrace
[02:04:33][SomeStep]    at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target)
[02:04:33][SomeStep] ##teamcity[buildStatus status='FAILURE' text='System.Exception: |'paket update nuget SomeNuget|' failed   at .... WEEEEEEEEEEEEEEEEEEEEEEEE
[02:04:33][SomeStep] 
[02:04:33][SomeStep] ---------------------------------------------------------------------
[02:04:33][SomeStep] Build Time Report
[02:04:33][SomeStep] ---------------------------------------------------------------------
[02:04:33][SomeStep] Target     Duration
[02:04:33][SomeStep] ------     --------
[02:04:33][SomeStep] Clean      00:00:00.0027612
[02:04:33][SomeStep] Clone      00:00:40.1434302
[02:04:33][SomeStep] Total:     00:03:42.5453691
[02:04:33][SomeStep] Status:    Failure
[02:04:33][SomeStep] ---------------------------------------------------------------------
[02:04:33][SomeStep]   1) System.Exception: 'paket update nuget SomeNuget' failed
[02:04:33][SomeStep]    at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1360.Invoke(String message)
[02:04:33][SomeStep]    at OurBuildFakeStackTrace
[02:04:33][SomeStep]    at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source)
[02:04:33][SomeStep]    at OurBuildFakeStackTrace
[02:04:33][SomeStep]    at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target)
[02:04:33][SomeStep] ---------------------------------------------------------------------
[02:04:34][SomeStep] Process exited with code 42
[02:04:34][SomeStep] Step Run OurBuildFakeScript.fsx (Command Line) failed

[forki]

I think it's already set to 180s in 5.2.1 (only reporting was wrong and fixed later)
what's the last version that worked?

[tebeco]

4.8.8 (sad i know)

v5.x works randomly depending of our infra response time
The issue is that the "Time to First Byte" is at leat 50sec on some source ... and we already know the nuget is NOT of that source

but we can't split to group because there's alway the need of nuget.org in their transitive

[forki]

/cc @matthid this can be the issue that we saw in #2465 - @tebeco I assume you need to pin to 4.x until @matthid gets a chance to look at it.

since you are using magic mode you want to put a paket.exe.config next to paket.exe - something like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="Version" value="..."/>
  </appSettings>
</configuration>

[tebeco]

We have a job running that re-install quite DL paket.bootstrapper.exe + rename + set it path as %PAKET_HOME% so all our build will works for sure without the need to commit the folder .paket

Do you think i can pin version 4.8.8 in paket.dependencies in magic mode without changing the existing paket.exe ?

[forki]

you can try. not sure if that works

[forki]

but you can just put that config file in paket_home folder

[tebeco]

well ... the webconfig might be a problem, i'll have to edit all our build first step that copy paket.exe to the .paket folder (dunno why ^^) but i'll give a try on what i can do here

[forki]

ok other idea - change the job to DL paket.bootstrapper.exe then instead of rename just call paket.bootstrapper.exe 4.xxx - so you have the correct version hanging around

[tebeco]

Any idea for #2465 estimate fix ?
I can managed with our nightly "Auto Update" build beeing red few days

The quickest / easy fix would be either

• in paket.dependencies
• as a command paramter of paket install/add/update
  A simply option (OPT OUT) to discard the timeout stuff ^^

A toggle feature OPT OUT

Else, as you said, i'll have to ask devs to step down from magic mode to pinned version + repush paket.lock on repositories

[forki]

I personally can't say. It's a bit tricky and I need help by @matthid

[tebeco]

On the same topic, in order to reduce the amount of useless network request
we have like 20 nuget from nuget.org and 5 to 15 internal nuget
and we got like 5 frequent nexus source for team nuget

So i'm already 100% SURE that we do 20 * 5 (100 request) to internal nexus that DOES NOT have any nuget (like FSharp.core ....) and will fail (take tiiimmmmmes to fail)

For example workaround would be instead of

source https://api.nuget.org/v3/index.json
source https://SomeInternal_TeamB_Nexus
source https://SomeInternal_TeamA_Nexus

nuget FSharp.Core
nuget Microsoft.Net.Compilers
nuget SomeInternal_TeamA_Nuget
nuget SomeInternal_TeamB_Nuget

doing :

source https://api.nuget.org/v3/index.json

nuget Microsoft.Net.Compilers
nuget SomeInternal_TeamA_Nuget source https://SomeInternal_TeamA_Nexus
nuget SomeInternal_TeamB_Nuget source https://SomeInternal_TeamB_Nexus

condition is that the transitive of A and B can also see source https://api.nuget.org/v3/index.json as a source

[tebeco]

(Do you think it would be interesting to create a ticket for that one ?)
'cause i think it would be a HUGE improuvment (well not for everyone ^^)

[forki]

source https://api.nuget.org/v3/index.json

nuget Microsoft.Net.Compilers

source https://SomeInternal_TeamA_Nexus
nuget SomeInternal_TeamA_Nuget

source https://SomeInternal_TeamB_Nexus
nuget SomeInternal_TeamB_Nuget 

^^ should work
A nuget dep should always only use sources above it's line

[tebeco]

soooo without creating group can i do this ?

source https://api.nuget.org/v3/index.json

nuget FSharp.Core
nuget Microsoft.Net.Compilers

source https://SomeInternal_TeamA_Nexus
nuget SomeInternal_TeamA_Nuget1
nuget SomeInternal_TeamA_Nuget2

source https://SomeInternal_TeamB_Nexus
nuget SomeInternal_TeamB_Nuget <== what happens if a transtive here is on https://SomeInternal_TeamA_Nexus

[forki]

what happens if a transtive here is on https://SomeInternal_TeamA_Nexus

it checks all sources above - so nuget.org and teamA as well

[tebeco]

well, once again something i did not know in paket :)
I'll try that work around, waiting for the OPT OUT (not IN i was wrong) for the timeout

[tebeco]

I managed to make thing worst, i'll give another try later today ;)

[tebeco]

I'll wait for the parameter INSIDE the paket.dependencies / command line instead of ENV var
(dependency on a var should be last hope not first ^^)

FYI : Found a tricky workaround so far

SET PAKET_RESOLVER_TASK_TIMEOUT=-1
Call_paket_here

the .Wait(-1) will disable the timeout :D

[matthid]

Please try again after #2499 is released
If possible make sure to switch to v3 nuget api if your internal nuget server support it.

Please report back. The environment variables are good for finding issues and testing performance, but really the default should work :)

[tebeco]

Closing
most of the issue comes from JFrog / Artifactory, i can't seem to make paket use their v3 url
the behavior change depending on the type of "repository" => Local (private) vs Remote (proxy to nuget.org)

Not sure if paket still does :
if(url.Contains('v3')) then
restoreV3
else
restoreV2

If so would benefits from an extension point like :
source https://..... feedVersion=v3
Nuget repositories URL cannot be trusted for something like this