server: TestDatabasesTablesV2 is flakey under testrace

[adityamaru]

Recently TestDatabasesTablesV2 has been flakey under testrace - https://teamcity.cockroachdb.com/project.html?projectId=Cockroach_Ci_Tests&testNameId=-5148906532967248544&tab=testDetails. All the failures seem to be off the form:

Error:      	Received unexpected error:
        	            	Get "https://127.0.0.1:33103/api/v2/databases/testdb/tables/public.testtable/": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestDatabasesTablesV2

Jira issue: CRDB-19138

[rafiss]

This is not a SQL experience test. Moving to SQL Observability.

[matthewtodd]

Before I understand anything here, on a gceworker, I can get ./dev test pkg/server --filter=TestDatabasesTablesV2 --stress --race --timeout=10m to pass by bumping the client timeout from 10 to 30 seconds:

--- a/pkg/rpc/tls.go
+++ b/pkg/rpc/tls.go
@@ -231,7 +231,7 @@ func (ctx *SecurityContext) GetUIServerTLSConfig() (*tls.Config, error) {
 // if needed. It uses the client TLS config.
 func (ctx *SecurityContext) GetHTTPClient() (http.Client, error) {
        ctx.lazy.httpClient.Do(func() {
-               ctx.lazy.httpClient.httpClient.Timeout = 10 * time.Second
+               ctx.lazy.httpClient.httpClient.Timeout = 30 * time.Second
                var transport http.Transport
                ctx.lazy.httpClient.httpClient.Transport = &transport
                transport.TLSClientConfig, ctx.lazy.httpClient.err = ctx.getUIClientTLSConfig()

I also notice that the failures, both historically and that I've observed in my own testing, have all been on the same endpoint, the databases/testdb/tables/public.testtable one.

Both of which lead me to believe that this endpoint could just be slow?

I'll dial down the timeout to see if I can find a reliable threshold (trying 20s now) and also start looking into the implementation to see if there's anything obvious there.

If not, options seem to be bumping the timeout, keeping the test skipped, or commenting out just the slow part.

[matthewtodd]

And, a clue, the first failures here happened in the development of #82617, which would corroborate the hypothesis that the endpoint just got slower. I'll look for the base commit there and try a git bisect to confirm.

[matthewtodd]

It's indeed the tableDetails endpoint that's now too slow for this test client's 10 second timeout.

While we could bump the timeout globally (unnecessarily lowering our signal for other slow endpoints), refactor to handle overriding the timeout on a request basis (less-than-straightforward with the http client library), or make a copy of the client with a higher timeout for this one request (current code would require dubious restructuring), I'm inclined to keep this test skipped until the underlying work in #84105 helps speed up the endpoint for real.

See also #90196.