Go development is generally slow when using github.com/uudashr/gopkgs

[codingconcepts]

I've found that when using the github.com/uudashr/gopkgs version of gopkgs, my CPU spikes for a long time and many on-save operations are extremely slow. If you look in your your Activity Monitor (on MacOS) or presumably Task Manager (on Windows if affected), you'll see gopkgs whirring away and not a lot happening in vscode.

I do have a pretty busy GOPATH (96 github.com subfolders etc.) but only open the top-level directory of the project I'm working on.

Since removing github.com/uudashr/gopkgs and replacing with github.com/tpng/gopkgs my IDE has come back to life but I keep getting prompted to install github.com/uudashr/gopkgs for the best experience. I can confirm that when I do heed the suggestion and re-install the uudashr version, my IDE grinds to a halt again.

[ramya-rao-a]

Thanks for reporting @codingconcepts!

We need both the import path and package name for the installed packages to provide certain features in the Go extension. github.com/tpng/gopkgs provides only the import path and so we made the move to github.com/uudashr/gopkgs

The results from github.com/uudashr/gopkgs gets cached, but the cache expires in 5 seconds. I can see how this can be aggressive in a busy GOPATH

What we can do is set the expiry to be the maximum between 5 seconds and the time it took to run gopkgs the previous time.

cc @uudashr

[ramya-rao-a]

@codingconcepts Can you try out the below version of the extension and see if it helps?

• Download https://github.com/Microsoft/vscode-go/blob/master/Go-latest.vsix
• Run code --install-extensions Go-latest.vsix
• If the above fails with Error: end of central directory record signature not found, then clone this repo and use the Go-latest.vsix file from the cloned repo
• Reload VS Code

[codingconcepts]

Hey Ramya, I installed the vsix without issue and have re-installed the two packages I've been ignoring updates for since discovering the performance improvement:

• github.com/ramya-rao-a/go-outline
• github.com/uudashr/gopkgs

Thanks for your help with this, I'll keep an eye on things today and let you know how I get on.

[codingconcepts]

Hey Ramya, vscode has become very slow again during the course of the day unfortunately. I've seen gopkgs consistently spiking to ~13% CPU and guru consistently spiking to ~350% CPU.

[ramya-rao-a]

Regarding guru, do you have codelens for references enabled? That calls guru frequently and guru is inherently slow on large code bases

Run gopkgs -format {{.Name}};{{.ImportPath}} in the terminal. How long does it take to complete?

Can you open ~/.vscode/extensions/lukehoban.go-0.6.73/out/src/goPackages.js and replace all occurrences of 5000 with 300000? This will make the cache expire every 5 mins.

[codingconcepts]

The results are a little misleading as the wall clock time is ~20s but here's a time result of gopkgs by itself:

gopkgs  1.36s user 1.95s system 14% cpu 22.070 total

I've updated the cache expiry of 5000 to 300000 in the file you've suggested and will get back to you later with the results, although I'd rather the cache was expiring every 5s, so everything's kept up-to-date obviously! While I do have a pretty big GOPATH, I shouldn't imagine it's uncommonly large, this slowness must be affected loads of people...

[ramya-rao-a]

The reason I suggested increasing the time is just narrow down what the issue really is. Is it just that gopkgs is taking too long or too many requests to gopkgs is being made.

But then I realized just now that we have put checks in place such that if a previous request to gopkgs is still running, we dont create a new request. We wait for the previous request to finish and use its results.

So increasing the expiry time is not going to do the trick. My guess is that even after you change to 300000 you wont see any difference. Do let me know if that is true.

This is about gopkgs itself taking up too many resources to complete a request.

[ramya-rao-a]

@codingconcepts Any updates?

[codingconcepts]

Hey ramya, sorry for the delay!

Unfortunately, things are still very very slow. I'll kick off a go run and it'll take around a minute, while gopkgs whirs away at around 2.5-3% CPU.

[codingconcepts]

It's also very slow on save (as I have [and want] linters running). Gopkgs spikes at around 7% CPU for about 30s.

[codingconcepts]

As you can imagine, GoToDefinition is also very slow too.

[uudashr]

@codingconcepts update gopkgs go get -u github.com/uudashr/gopkgs/cmd/gopkgs
What is the result of time gopkgs |wc -l?

and also time gopkgs -workDir=$(go list -f "{{.Dir}}" github.com/uudashr/gopkgs)|wc -l?

[codingconcepts]

Hiya,

Here's the output for the first command:

6103
gopkgs  2.06s user 3.18s system 18% cpu 28.842 total
wc -l  0.00s user 0.00s system 0% cpu 28.838 total

And the second command:

1907
gopkgs -workDir=$(go list -f "{{.Dir}}" github.com/uudashr/gopkgs)  0.70s user 1.05s system 19% cpu 8.930 total
wc -l  0.00s user 0.00s system 0% cpu 8.927 total

[uudashr]

Here mine,

    6141
gopkgs  2.20s user 3.22s system 104% cpu 5.210 total
wc -l  0.00s user 0.00s system 0% cpu 5.209 total

and

     646
gopkgs -workDir=$(go list -f "{{.Dir}}" github.com/uudashr/gopkgs)  0.26s user 0.44s system 115% cpu 0.611 total
wc -l  0.00s user 0.00s system 0% cpu 0.610 total

Returned package numbers not too much different for the first command, but the difference on time taken are so significant.

Not sure why. But based on the discussion, is it possible there is a problem on your go/mac installation? Seems everything is slower

[uudashr]

Please tell me if there are any progress. Currently I have no idea.

[klauern]

I'm running on Windows, and I am having some significant perf issues myself. Unfortunately, the tests that are provided are more bash-centric, as I don't believe $(go ...) commands can be executed within cmd.exe or powershell.exe, so I don't have an easy way to run these tests myself.

Is there an equivalent call that can work on Windows 10?

[uudashr]

Not too familiar with Win 10.
You can try to execute go list -f "{{.Dir}}" github.com/uudashr/gopkgs
and put the result on the workDir param gopkgs -workDir=<location-of-gopkgs-src-dir>

[klauern]

Okay, I was able to find a way to measure performance using Powershell with the Measure-Command call:

Now, with the tpng version:

[uudashr]

Around 200ms fast enough.
Can you do gopkgs without -workDir

[klauern]

Ah, I think I did something wrong. I may have not built the uudashr/gopkgs version correctly, instead accidentally building the tpng version twice...

Here's the corrected output, also using go get to verify that I did install the right version:

[klauern]

And here's the corrected output from tpng's version:

[uudashr]

Can you show me how many packages returned on uudashr/gopkgs
Try https://superuser.com/questions/959036/what-is-the-windows-equivalent-of-wc-l for line count

[klauern]

Here's what I ran

go get -u github.com/uudashr/gopkgs/cmd/gopkgs
(gopkgs.exe | Measure-Object -line).Lines
go get -u github.com/tpng/gopkgs
(gopkgs.exe | Measure-Object -line).Lines

[uudashr]

Well... 6 seconds for 1325 packages is way too long.
I don't have access to Win machine right now. btw this is gopkgs issue on Win os.

[codingconcepts]

Another update from my long-suffering CPU! I noticed a fair amount of fan action this morning and found this (it continued for over a minute):

[codingconcepts]

Would it be possible to remove the extension prompt that requires the uudashr version of gopkgs? I've reinstalled the tpng version because - performance wise - my machine agrees with it and I might consider either updating that to provide package names or writing my own. I just don't want to be prompted constant to upgrade to something that slows everything down for me:

[uudashr]

We need better gopkgs... not just about performance, but about the functionality.
tngp give better performance, but it does not have features required by vscode.

I've been experiencing this, long ago. But no more after couple fix. Not sure why it hung.. idle wake ups seems too long. Can be on vscode-go code (the caller) or on the gopkgs. Help are welcome.

For now try to kill the gopkgs.

@codingconcepts your runtime is Go 1.7? If yes probably I need to test using the 1.7 version.

[ramya-rao-a]

@codingconcepts, @klauern There have been some improvements that @uudashr has done to the the gopkgs tool to make it faster. Can you please try the below and let us know if the changes make any difference to your case?

• Download https://github.com/Microsoft/vscode-go/blob/master/Go-latest.vsix
• Run code --install-extension Go-latest.vsix
• If the above fails with Error: end of central directory record signature not found, then clone this repo and use the Go-latest.vsix file from the cloned repo
• Reload VS Code

[codingconcepts]

Hey Ramya, sorry for the delay!

I've installed the latest version of the Go extension and updated to the latest version of gopkgs when prompted and so far, everything seems to be behaving.

I'll keep an eye on things during the day and get back to you :)

[codingconcepts]

Hey @ramya-rao-a,

I've had some time to play with the new version and would like to report that the general experience of using Go has definitely improved.

I'm noticing much higher than normal CPU usage for codehelper now though. It's spinning at around 120% core's worth of CPU, which wasn't happening before the upgrade.

I've killed the process for now but if it happens again, let me know what kind of metrics you'd like me to grab and I'll grab them for you.

Cheers,

Rob

[ramya-rao-a]

@codingconcepts Follow the wiki on Performance Issues. That should help you figure out on what's making the codehelper spin like that.

[ramya-rao-a]

@codingconcepts The fix @uudashr made is now out in the latest update to the Go extension (0.6.81)
I'll close this issue for now based on your feedback in #1490 (comment)

If you find anything while investigating the high CPU usage, please log a new issue.

And thanks a lot for your patience and trying out the fixes that we are putting out.

Happy Coding!