Skip to content

Commit

Permalink
roachtest: stabilize tpccbench/nodes=3/cpu=16
Browse files Browse the repository at this point in the history
Fixes cockroachdb#61973. With tracing, our top-of-line TPC-C performance took a
hit. Given that the TPC-C line searcher starts off at the estimated max,
we're now starting off at "overloaded" territory; this makes for a very
unhappy roachtest.

Ideally we'd have something like cockroachdb#62010, or even admission control, to
not make this test less noisy. Until then we can start off at a lower
max warehouse count.

This "fix" is still not a panacea, the entire tpccbench suite as written
tries to nudge the warehouse count until the efficiency is sub-85%.
Unfortunately, with our current infrastructure that's a stand-in for
"the point where nodes are overloaded and VMs no longer reachable". See
\cockroachdb#61974.

---

A longer-term approach to these tests could instead be as follows.
We could start our search at whatever the max warehouse count is (making
sure we've re-configure the max warehouses accordingly). These tests
could then PASS/FAIL for that given warehouse count, and only if FAIL,
could capture the extent of the regression by probing lower warehouse
counts. This is in contrast to what we're doing today where we capture
how high we can go (and by design risking going into overload territory,
with no protections for it).

Doing so lets us use this test suite to capture regressions from a given
baseline, rather than hoping our roachperf dashboards capture
unexpected perf improvements (if they're expected, we should update max
warehouses accordingly). In the steady state, we should want the
roachperf dashboards to be mostly flatlined, with step-increases when
we're re-upping the max warehouse count to incorporate various
system-wide performance increases.

Release note: None
  • Loading branch information
irfansharif committed Mar 15, 2021
1 parent 97f40dc commit 72c96fa
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pkg/cmd/roachtest/tpcc.go
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ func registerTPCC(r *testRegistry) {
CPUs: 16,

LoadWarehouses: gceOrAws(cloud, 2500, 3000),
EstimatedMax: gceOrAws(cloud, 2200, 2500),
EstimatedMax: gceOrAws(cloud, 2100, 2500),
})
registerTPCCBenchSpec(r, tpccBenchSpec{
Nodes: 12,
Expand Down

0 comments on commit 72c96fa

Please sign in to comment.