Theia's requests to open-vsx.org lead to huge responses

[mbarbero]

The Eclipse Foundation is operating open-vsx.org and we noticed that the requests POST /api/-/query (which are Theia specifics AFAIK), leads to 2.5+MB of gzipped json responses. This consume a large chunk of the Eclipse Foundation bandwidth to the detriment of other services / projects.

Would it be possible to reduce the amount of data retrieved by those requests? Would it need some changes on the openvsx server to be more efficient?

[vince-fugnitto]

@mbarbero thank you for the discussion!

we noticed that the requests POST /api/-/query (which are Theia specifics AFAIK), leads to 2.5+MB of gzipped json responses.

I don't believe the post endpoint for query is used any longer (#10325).

[msujew]

I imagine that a lot of users still use older versions of Theia (#10325 was only included in 1.19). Ideally during the next few months, people update their Theia installations and start using the GET endpoint, which is caching everything, reducing the load drastically. However, I'm not sure whether using cached responses reduces the bandwidth-load considerably.

[spoenemann]

The amount of data returned by the GET endpoint is the same, so we could have a closer look what information we really need and add additional options to control what is included in the response.

[amvanbaren]

I can add offset and size query parameters to add paging to the /api/-/query endpoint. The /api/-/search endpoint has the same parameters to limit the response size. offset and totalSize are added to the query response, so that you can page through the extensions.

On the Theia side you then need to implement paging for the getAllVersions function.

theia/dev-packages/ovsx-client/src/ovsx-client.ts

Line 89 in bf00fd0

async getAllVersions(id: string): Promise<VSXExtensionRaw[]> {