install/upgrade/outdated - improve performance and optimize queries

[wickles]

Don't know if this is expected behavior or it's just my system, but choco outdated always takes a good 45 seconds to run fully, with 136 packages installed. It seems like it's querying for new data for each package, every time. Most package managers have a mypacman update style command to update a local cache of complete package data all at once, so that we don't need to query every package every time, and this command itself is generally much quicker than choco outdated for other package managers. It would be great if we could get a similar feature or some other way to speed up choco outdated.

---

NOTE: Added by admin.

Seeing Errors in New Versions of Choco?

If you find this causes issues with your repository of choice, please see #1770

Known Affected Servers include LaGet and NuGet Simple Server (PHP). Please see #1770 for those items as that is where that will be updated.

Release Notes

So we've implemented some optimizations for install/upgrade/outdated on getting packages. It is largely seen in outdated, where it is an order of magnitude faster (but it also has the most queries).

For Chocolatey v0.10.12+, we have converted to a much more efficient way of finding packages that allows us to reduce bandwidth and really speed things up. To do so, it means a change from FindPackagesById() to Packages() with a filter when we have a DataServicePackageRepository (aka a NuGet OData Package Repository Atom Feed).

The prior method FindPackagesById() would return ALL versions of a package and then locally choco/NuGet would filter them. This as you know can cause multiple queries if you have a lot of versions of a package, and some really inefficient use of resources and time. Moving to Packages() filtering by package Id and the latest version available means 1 result is returned. It's quite efficient and quite a bit faster than the older method by an order of magnitude.

Outdated Measurements

choco outdated makes heavy use of calling for upgraded versions. I have a system with about 34 packages installed.

• Chocolatey 0.10.11 - Measure-Command { choco outdated -r --ignore-unfound }

  TotalSeconds       : 5.4484086
  TotalMilliseconds : 5448.4086
  

  • Roughly 160 milliseconds per package

• Chocolatey 0.10.13 - Measure-Command { choco outdated -r --ignore-unfound }

  TotalSeconds       : 2.6337265
  TotalMilliseconds : 2633.7265 
  

  • Roughly 77 milliseconds per package

That's over 100% faster if I can math correctly.

[ferventcoder]

@wickles yes, it's on our list to implement package indexes. That is why when you run choco update you will get a deprecated message that mentions that the command is being deprecated and will come back for that.

[ferventcoder]

If you think it's bad now, you never had to work with the old Posh version of Choco - it was more like 40 minutes. Really, really bad. We've been talking about adding package indexes since 2014. https://www.slideshare.net/ferventcoder/chocolatey-puppet-conf2014/27

[dionysius]

Am I allowed to kindly be vocal about this slowiness here? It bugs me quite some time every time I use choco, any command it is soooo slow :(

C:\Users\Dionysius
λ ptime choco upgrade chocolatey

ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes <[email protected]>

=== choco upgrade chocolatey ===
Chocolatey v0.10.8
Upgrading the following packages:
chocolatey
By upgrading you accept licenses for the packages.
chocolatey v0.10.8 is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Enjoy using Chocolatey? Explore more amazing features to take your
experience to the next level at
 https://chocolatey.org/compare

Execution time: 86.252 s

[ferventcoder]

@dionysius head to the logs folder and take a look at the timing of the output of each log message. Even better, post it to a gist and share that here. That will help us see what may be causing things to be slow for you. That is a very long time to be waiting on a command and definitely not normal. It's not quite the same as this issue, as choco outdated is network dependent and needs to make quite a few calls, but most commands should be done quite quickly.

[ferventcoder]

The chocolatey.log file by the way, not the summary.

[dionysius]

Hi @ferventcoder, didn't expected that quick response. While my issue is a general issue, I'll better raise a separate issue for that.

But maybe I can help a little for the time beeing, I can contribute to this exact named issue I've encounter also for choco outdated:

λ ptime choco outdated

ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes <[email protected]>

=== choco outdated ===
Chocolatey v0.10.8
Outdated Packages
 Output is package name | current version | available version | pinned?

npcap|0.83|0.83|false

Chocolatey has determined 0 package(s) are outdated.
 1 package(s) had warnings.
Warnings:
 - npcap

Execution time: 107.479 s

(I'm aware why npcap isn't listed anymore, there's a thread for it elsewhere)

The gist of chocolatey.log: https://gist.github.com/dionysius/c973f254355b04d349e6c9124993e89a

And in theory my computer and network is not potato :)

[bcurran3]

It takes 45-48 seconds for choco outdated to give me results as well.

I've noticed a considerable slowdown in choco processing after the version header for some time. I believe it goes back to when the manifest issue was changed back to not requiring elevated rights.

[dionysius]

Oh, good to know. All my commands are in elevated cmd already, otherwise the "auto confirmation" would not apply. (I think it's the option called "allowGlobalConfirmation", but It was long time ago I activated that)

Edit: And maybe it is how it is then - if so, I just like to let you know that there are wishes for improvement :)

[bcurran3]

I retract my statement about all choco command being slow. There was a point that it seemed that way, but I think it went away with 10.7 or 10.8.

@dionysius - thanks for the clue on ptime. I like it. I used to use the timer function in 4DOS a lot and needed a replacement since I haven't used 4DOS in years.

just for comparison

ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes [email protected]

=== choco outdated ===
Chocolatey v0.10.8 Professional
Outdated Packages
Output is package name | current version | available version | pinned?

irfanviewshellextension|1.04|1.04|false
nirlauncher|1.20.16|1.20.17|false
realtek-audio-drivers|2.82|2.82|false

Chocolatey has determined 1 package(s) are outdated.
2 package(s) had warnings.
Warnings:

• irfanviewshellextension
• realtek-audio-drivers

Execution time: 78.027 s

[Ytrog]

I only have 34 packages and choco outdated is also very slow for me. It used to be fast however. Nowadays I sometimes even have a timeout :(

[ferventcoder]

Choco outdated is always going to be dependent on the network, at least until pkg indexes are implemented and even then in some way when the indexes update.

[ferventcoder]

This has been completed to speed up queries - this is available now in the beta (choco upgrade chocolatey --pre) and will be in 0.10.12.

[dionysius]

Can see the improvement :)

λ ptime choco outdated

ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes <[email protected]>

=== choco outdated ===
Chocolatey v0.10.12-beta-3-gb65f586
Outdated Packages
 Output is package name | current version | available version | pinned?

[...]

Chocolatey has determined 10 package(s) are outdated.

Execution time: 68.250 s

Compared to my earlier try (should be the same package list) it has at almost halved 👍 (107s down to 68s)

[wickles]

Nice. Thanks for the update!

[asheroto]

I know this is an old issue but it raises some questions...

• Why does Chocolatey query each package one by one? Why not concurrently?
• Why does the entire package information have to transfer for commands such as choco outdated?
• Why not have one query to the server that requests version numbers for all requested packages?

Currently choco outdated will issue GET requests one-by-one such as...

/api/v2/Packages()?$filter=(tolower(Id) eq 'audacity') and IsLatestVersion
/api/v2/Packages()?$filter=(tolower(Id) eq 'autohotkey') and IsLatestVersion
/api/v2/Packages()?$filter=(tolower(Id) eq 'ccleaner') and IsLatestVersion

Instead, consider one consolidated request such as...
/api/v2/Packages()?$filter=('audacity','autohotkey','ccleaner')

Example returned information:

{
  "audacity": "3.0.5",
  "autohotkey": "1.1.33.10",
  "ccleaner": "5.86.9258"
}

I realize this would take some additional work on the server side, but it would

1. Save a lot of traffic
2. Be much faster for users because only 1 GET request would take place

In addition, the amount of data returned by the API seems unnecessary for simple operations such as choco outdated.

Example query - 6 KB of data

Considering each request returns an average of 6 KB of data...

I have 193 packages... so it transfers 1.158 MB, which isn't a huge deal in terms of size, but having to process 193 GET requests of 6 KB each...takes a little bit, and for certain functions such as choco outdated, we don't need most of the data that is returned in each request.... we just need the version number.

Even if the team didn't want to implement a multi-package search in the API, an alternative solution would be:

• Return only the package version when requested
• Run the GET commands concurrently through async methods

Perhaps there is a backstory I am unaware of, or perhaps this is available in the business version (I only use it for personal use), but I happened to be Googling about this topic and came across this issue.

Thoughts?

Thank you guys for all the work you do on Chocolatey!

[dionysius]

Fell free to hide this comment as this won't contribute to the issue.

I was forced to switch to another package manager called scoop because of this and I'm very happy with it.

[pauby]

@asheroto This issue is closed. Can you please open a new issue.

[asheroto]

No prob! 🙂

[UsernameIsAlreadyTakenByMe]

I'm not sure which issue is solved now, but the "choco upgrade all" command became really fast in "chocolatey v2.2.0" :)
...I just wanted to give some hope for the people using google and come here :)

Thank you :)

[pauby]

Performance improvements were made in 2.1.0 and this issue helps choco upgrade all for 2.2.0.