x/tools/gopls: improvements to loading behavior #68002
Comments
gopherbot
added
Tools
|
Similar Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.) |
|
I'm responsible for a system that generates a lot of rpc automated code that exists as a remote package that is slow to load and slow to compile. I'm currently working on a GoPackageDriver to automatically generate the code locally at the user's site, which would help a lot if the speed could be increased. ❤️ |
|
Change https://go.dev/cl/611843 mentions this issue: |
One of the unsolved problems of the gopls scalability redesign described at https://go.dev/blog/gopls-scalability is that concurrent type checking operations may perform redundant work. While we avoided redundancy within the context of *one* operation, by encapsulating the type checking batch, it is possible that there will be logically distinct operations occurring concurrently, such as computing diagnostics and autocompletion. These operations could not share type information, as they were operating within distinct type checking "realms" (recall that go/types relies on the canonicalization of imports). This CL addresses that problem by refactoring the type checking batch such that it can join two distinct operations into a shared state. The typeCheckBatch becomes a queryable entity, coordinating work across ongoing package graph traversals. This required surprisingly little change to the typeCheckBatch itself, which already abstracted the notion of a package traversal. Rather, the key missing component was a future cache implementation that was (1) retryable and (2) transient, in the sense that computed values were discarded after use. The bulk of this change is the implementation and testing of such a cache. Some elements of the refactoring remain awkward: - packageHandles are collected and merged into a shared map, because they must still be computed in large batches for efficiency. - the shared importGraph remains a clumsy optimization, requiring subtle logic to pre-build a shared import graph. My intuition is that both of these problems have an elegant solution, but this work is left for a subsequent CL. In addition to reducing CPU across the board, due to less redundancy in the existing diagnostic pass, this CL will facilitate the following additions to gopls: - Pull-based diagnostics (golang/go#53275): now that diagnostic operations are naturally joined, there is less need for gopls to manage the scheduling of the diagnostic pass. - Improvements to loading behavior (golang/go#68002): being able to handle requests asynchronously allows completion requests to potentially be handled using stale metadata. Updates golang/go#53275 Updates golang/go#68002 Change-Id: Ib228cdcce2c4b6f616d6ba5b0abeb40e87f449be Reviewed-on: https://go-review.googlesource.com/c/tools/+/611843 Reviewed-by: Alan Donovan <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>
|
I got this working, and it was quite a bit harder than expected. Furthermore, the change in loading behavior has approximately no impact on performance when used with
Therefore, I'm not sure whether this is worth doing. It's a lot of complexity for little gain. I think the more impactful change is to be able to run completion on stale metadata: I propose we skip straight to that. |
|
This issue aggregates some work I'd like to do to refine the
go/packages.Loadcalls made by gopls. Generally speaking, these fall into the category of "using gopls' knowledge of the workspace state to refine loading patterns". This should benefit users across the board, but may in particular help for users of go/packages drivers that have different performance characteristics than the go command.A word on terminology: we refer to the packages data returned by
go/packages.Loadas package metadata.Considering only the simple case of a module workspace, metadata loading works approximately as follows (some edge cases are ignored). All loads are inclusive of
-deps.<modulePath>/...file=query (but keep track of files that can't be loaded so we don't keep trying).This loading pattern evolved to ensure we don't miss metadata changes, and can operate on a file as soon as possible. Notably, it was only chosen based on the go command behavior, and generally works because
go listis heavily optimized. Even loading a relatively large project such as Kubernetes takes only a few seconds.However, there are some improvements to be desired:
-depswhen we're just selectively reloading a file or package. We can theoretically instead load without-deps, see which imports (if any) are missing, and then load those missing imports in a second pass.<modulePath>/....We should experiment with these improvements. I believe they should be of particular help for users of a bazel go/packages drivers, as bazel queries have significantly higher overhead than the go command.
CC @adonovan @JamyDev
The text was updated successfully, but these errors were encountered: