really bad performance using AppVeyor feeds with > 2.17.3 #1317

ctaggart · 2015-12-19T05:53:20Z

We have 60 dependencies in a paket.depdendencies with 55 of them from NuGet Gallery and 5 of them our AppVeyor feed. Paket 2.17.3 took 45 seconds to do a paket update. Paket 2.18.0 took 9 minutes.

Paket 2.17.3 will tries 1 request per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/package-versions/Auth0?includePrerelease=true HTTP/1.1
Authorization: Basic 
Accept: application/atom+json,application/json
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:26:07 GMT
Content-Length: 2

[]

Paket 2.18.0 and above makes 4 requests per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20NormalizedVersion%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Length: 290
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:04 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">No property 'NormalizedVersion' exists in type 'Appveyor.Models.Package' at position 18.</message>
</error>

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20Version%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic 
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 565
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/8.5
DataServiceVersion: 2.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
  <title type="text">Packages</title>
  <id>https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/Packages</id>
  <updated>2015-12-19T05:29:05Z</updated>
  <author>
    <name />
  </author>
  <link rel="self" title="Packages" href="Packages" />
</feed>

GET https://ci.appveyor.com/nuget/ourfeed/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic 
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Content-Length: 248
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">Resource not found for the segment 'Packages'.</message>
</error>

GET https://ci.appveyor.com/nuget/ourfeed/odata/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic 
Accept-Charset: UTF-8
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml, application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: private
Content-Length: 245
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:06 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">Resource not found for the segment 'odata'.</message>
</error>

The text was updated successfully, but these errors were encountered:

forki · 2015-12-19T07:42:46Z

Thanks for reporting this.

forki · 2015-12-19T08:00:46Z

Ok the fix that I have in mind will hopefully make 0 calls. Let me see if I
find some time on Monday.
On Dec 19, 2015 06:53, "Cameron Taggart" [email protected] wrote:

We have 60 dependencies in a paket.depdendencies with 55 of them from
NuGet Gallery and 5 of them our AppVeyor feed. Paket 2.17.3 took 45 seconds
to do a paket update. Paket 2.18.0 took 9 minutes.

Paket 2.17.3 will tries 1 request per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/package-versions/Auth0?includePrerelease=true HTTP/1.1
Authorization: Basic
Accept: application/atom+json,application/json
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:26:07 GMT
Content-Length: 2

[]

Paket 2.18.0 and above makes 4 requests per missing dependency to AppVeyor:

GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20NormalizedVersion%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Length: 290
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:04 GMT
No property 'NormalizedVersion' exists in type 'Appveyor.Models.Package' at position 18.
GET https://ci.appveyor.com/nuget/ourfeed/Packages?$filter=Id%20eq%20'Auth0'%20and%20Version%20eq%20'1.11.3' HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 565
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/8.5
DataServiceVersion: 2.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT
<title type="text">Packages</title> https://ci.appveyor.com/nuget/tachyuscorp-cdj2r7dkqyo6/Packages 2015-12-19T05:29:05Z
GET https://ci.appveyor.com/nuget/ourfeed/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept: application/atom+xml,application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Content-Length: 248
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:05 GMT
Resource not found for the segment 'Packages'.
GET https://ci.appveyor.com/nuget/ourfeed/odata/Packages(Id='Auth0',Version='1.11.3') HTTP/1.1
Authorization: Basic
Accept-Charset: UTF-8
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml, application/xml
User-Agent: Paket
Host: ci.appveyor.com

HTTP/1.1 404 Not Found
Cache-Control: private
Content-Length: 245
Content-Type: application/xml
Server: Microsoft-IIS/8.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 19 Dec 2015 05:29:06 GMT
Resource not found for the segment 'odata'.
—
Reply to this email directly or view it on GitHub
#1317.

…r a package - references #1317

forki · 2015-12-19T16:04:58Z

#1319 should make things much better. I hope I find good Internet connection this evening to release a prerelease.
But review would be very welcome.

forki · 2015-12-19T18:02:28Z

Ok a prerelease version is now available. Please let me know if it works for you.

// @vasily-kirichenko I remember you also had more than one feed and saw slower updates. Maybe this helps for you too.

There is still one new issue open to make "paket install" faster as well. Hopefully find some time on Monday.

vasily-kirichenko · 2015-12-19T18:12:23Z

I have very slow "install" since 2.24.0, updates are even slower.

forki · 2015-12-19T18:13:29Z

I will check that too
On Dec 19, 2015 7:12 PM, "Vasily Kirichenko" [email protected]
wrote:

I have very slow "install" since 2.24.0, updates are even slower.

—
Reply to this email directly or view it on GitHub
#1317 (comment).

ctaggart · 2015-12-19T22:47:40Z

Thank you. I will test this when I'm in the office on Monday.

forki · 2015-12-20T12:20:46Z

@vasily-kirichenko please check the new alpha version if you find some time. It does some things a little bit smarter. I hope it helps.

…r a package - references #1317

forki · 2015-12-21T14:26:42Z

since I also noticed a caching issue I had to invalidate all cache files.

ctaggart · 2015-12-22T03:54:44Z

This is not fixed in 2.37.1. There are still 3 requests that fail after the first call returns an empty list of packages. There are 3 extra unnessary failed requests for each package. The good news is that the paket update was only 2 minutes & 23 seconds. It looks like the cache fix helped.

forki · 2015-12-22T07:18:49Z

What calls?
On Dec 22, 2015 04:54, "Cameron Taggart" [email protected] wrote:

This is not fixed in 2.37.1. There are still 3 requests that fail after
the first call returns an empty list of packages. There are 3 extra
unnessary failed requests for each package. The good news is that the paket
update was only 2 minutes & 23 seconds. It looks like the cache fix
helped.

—
Reply to this email directly or view it on GitHub
#1317 (comment).

forki · 2015-12-22T12:20:57Z

Ok applied another fix. Did it help?

ctaggart · 2015-12-22T18:12:43Z

Yes! There is only one call per package to AppVeyor now and the responses are all HTTP 200s. The paket update time dropped to 51 seconds for the same project with version 2.38.0.

Related, but not required for this ticket:

3 requests are being made to nuget.org for each package that it doesn't have. #1320 would make this irrelevant for me.

forki · 2015-12-23T13:48:53Z

Could you please retry with 2.39.1 (twice)? The second update run should only use the correct feeds.

vasily-kirichenko · 2015-12-24T07:37:39Z

2.39.1.0: 40 seconds first run, 15 seconds subsequent runs. Much faster, still slower than 2.23 (6 seconds).

forki added a commit that referenced this issue Dec 19, 2015

Only check PackageDetails for Sources that responded with versions fo…

45db71f

…r a package - references #1317

forki mentioned this issue Dec 19, 2015

Only check PackageDetails for Sources that have versions #1319

Merged

forki added a commit that referenced this issue Dec 19, 2015

Only check PackageDetails for Sources that responded with versions fo…

6e5ec1a

…r a package - references #1317

ctaggart mentioned this issue Dec 19, 2015

ability to specify which sources to pull the dependencies from #1320

Closed

forki added a commit that referenced this issue Dec 20, 2015

Only check PackageDetails for Sources that responded with versions fo…

4b768d4

…r a package - references #1317

ctaggart closed this as completed Dec 22, 2015

ctaggart mentioned this issue Dec 22, 2015

can't restore packages from AppVeyor feed in 2.38.0 #1326

Closed

ctaggart mentioned this issue Dec 23, 2015

Cache which source contains versions in GetVersions #1327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

really bad performance using AppVeyor feeds with > 2.17.3 #1317

really bad performance using AppVeyor feeds with > 2.17.3 #1317

ctaggart commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

vasily-kirichenko commented Dec 19, 2015

forki commented Dec 19, 2015

ctaggart commented Dec 19, 2015

forki commented Dec 20, 2015

forki commented Dec 21, 2015

ctaggart commented Dec 22, 2015

forki commented Dec 22, 2015

forki commented Dec 22, 2015

ctaggart commented Dec 22, 2015

forki commented Dec 23, 2015

vasily-kirichenko commented Dec 24, 2015

really bad performance using AppVeyor feeds with > 2.17.3 #1317

really bad performance using AppVeyor feeds with > 2.17.3 #1317

Comments

ctaggart commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

forki commented Dec 19, 2015

vasily-kirichenko commented Dec 19, 2015

forki commented Dec 19, 2015

ctaggart commented Dec 19, 2015

forki commented Dec 20, 2015

forki commented Dec 21, 2015

ctaggart commented Dec 22, 2015

forki commented Dec 22, 2015

forki commented Dec 22, 2015

ctaggart commented Dec 22, 2015

forki commented Dec 23, 2015

vasily-kirichenko commented Dec 24, 2015