build: bench running time

[herkolategan]

Running time of bench (part of Extended CI on TeamCity) has increased from an average of 60 minutes to 90 minutes around 5 July 2023.

This issue is to track an investigation as to whether we should increase the timeout, which is 90 minutes currently, or investigate the source of the longer running time.

A basic analysis points to #106177 as the PR that introduced the longer running time. And an un-merged draft build-test PR negating the change #106400 where the running time is back to 60 minutes again.

Jira issue: CRDB-29627

[blathers-crl]

cc @cockroachdb/test-eng

[jbowens]

It seems to be driven by a regression in batcheval.BenchmarkRefreshRange. I don't have bandwidth to investigate at the moment. Adding to the Storage board.

[srosenberg]

Running time of bench (part of Extended CI on TeamCity) has increased from an average of 60 minutes to 90 minutes around 5 July 2023.

That's a pretty large increase. @herkolategan Is there an easy way to see the set of (individual) benches which were impacted? It also seems CI (on master) is back to ~60 minute runs as of yesterday; was there a fix?

[jbowens]

s there an easy way to see the set of (individual) benches which were impacted?

Many inspection revealed it was all batcheval.BenchmarkRefreshRange.

is back to ~60 minute runs as of yesterday; was there a fix?

Ah, I think that explains it. I'm pretty confident we observed this issue: #106672 (comment)

#106177 enabled a new sstable table format, which allowed this new block property to begin to be recorded. A fix for the bug was merged in #106923 yesterday. (cc @sumeerbhola)

I'm going to close this out.