[ws-proxy] tune idle connection pool

[akosyakov]

What it does

By default the connection pool is capped only by 2 connections per host.
If a client tries to open many connections then latency increased significantly, see https://stackoverflow.com/a/60114945/961588. This commit configures the connection pool be unlimited but capped by 100 connections per host. It was inspired by measurement in Caddy: caddyserver/caddy#2805

I tried different values:

• 32 was giving a result over 1mins with cached data
• 100 was giving a result around 20s with cached data
• 1000 was giving a result around 30s with cached data

How to test

• Start a workspace for https://ak-proxy-test.staging.gitpod-dev.com/#https://github.com/gitpod-io/vscode
• Wait till VS Code compiles and tab on port 3000 opened.
• Open 3000 port first time and wait that everything is cached.
• After that reload several times again and check that each time VS Code pops up and requests does not time out.

We should also have some load testing.

[akosyakov]

Setting 0 and 32 did help, but it is slow. I would play a bit more with Caddy and ws-proxy settings to see whether it can be faster.

[akosyakov]

It comes from 1 mis 20s to just 20s if 100 connections per host are allowed, let's see if 1000 makes bigger difference.

Updated: 1000 actually makes everything slower again, will stick with 100.

[codecov]

Codecov Report

Merging #4461 (4054d62) into main (2588d5f) will increase coverage by 67.32%.
The diff coverage is 50.00%.

❗ Current head 4054d62 differs from pull request most recent head 039943a. Consider uploading reports for the commit 039943a to get more accurate results

@@            Coverage Diff            @@
##           main    #4461       +/-   ##
=========================================
+ Coverage      0   67.32%   +67.32%     
=========================================
  Files         0        7        +7     
  Lines         0      808      +808     
=========================================
+ Hits          0      544      +544     
- Misses        0      214      +214     
- Partials      0       50       +50     

Impacted Files	Coverage Δ
components/ws-proxy/pkg/proxy/config.go	0.00% <0.00%> (ø)
components/ws-proxy/pkg/proxy/pass.go	73.07% <100.00%> (ø)
components/ws-proxy/pkg/proxy/workspacerouter.go	74.19% <0.00%> (ø)
components/ws-proxy/pkg/proxy/auth.go	100.00% <0.00%> (ø)
components/ws-proxy/pkg/proxy/routes.go	86.88% <0.00%> (ø)
components/ws-proxy/pkg/proxy/cookies.go	73.33% <0.00%> (ø)
components/ws-proxy/pkg/proxy/infoprovider.go	48.54% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2588d5f...039943a. Read the comment docs.

[akosyakov]

FYI: I'm using this branch to work on VS Code please don't merge. I don't have other env which allow to open VS Code from sources reliably.

[geropl]

It comes from 1 mis 20s to just 20s if 100 connections per host are allowed, let's see if 1000 makes bigger difference.
Updated: 1000 actually makes everything slower again, will stick with 100.

Also, this is hard to judge in preview envs because there we have 3 proxies in a row: staging proxy -> proxy -> ws-proxy
We'd have to measure in staging on a fat/ws-cluster and compare the result to have sth reliable.

[akosyakov]

@geropl How should we proceed then? It can be load tested only on staging?

[geropl]

Works as expected. Takes more than 20s with my connection, but absolutely no timeouts whatsoever. 👍

@akosyakov IMO we should merge as we're lagging the prober tools (per-branch staging) to test this, and the changes are reasonable. We can still test and fine-tune on staging if necessary.