Skip to content

Commit

Permalink
Merge #81565 #82382 #82740 #82744 #82748
Browse files Browse the repository at this point in the history 
81565: roachtest: benchmark node decommission r=AlexTalks a=AlexTalks

roachtest: benchmark node decommission

While previously some roachtests existed for the purposes of
testing the decommission process, we have not had any benchmarks to
track how long it takes to decommission a node, making it difficult to
reason about how to understand what makes decommission so slow. This
change adds benchmarks for node decommission under a number of
configurations, including variable numbers of nodes/cpus, TPCC
warehouses, and with admission control enabled vs. disabled.

Some initial runs of the test have shown the following averages:
```
decommissionBench/nodes=4/cpu=16/warehouses=1000: 16m14s
decommissionBench/nodes=4/cpu=16/warehouses=1000/no-admission: 15m48s
decommissionBench/nodes=4/cpu=16/warehouses=1000/while-down: 20m36s
decommissionBench/nodes=8/cpu=16/warehouses=3000: 18m30s
```

Release note: None

82382: kvstreamer: optimize singleRangeBatch.Less r=yuzefovich a=yuzefovich

**bench: add benchmarks for lookup joins**

This commit adds benchmarks for lookup joins, both when equality columns
are and are not key, both with and without maintaining ordering.

Release note: None

**kvstreamer: optimize singleRangeBatch.Less**

This commit optimizes `singleRangeBatch.Less` method which is used when
sorting the requests inside of these objects in the OutOfOrder mode
(which is needed to get the low-level Pebble speedups) by storing the
start keys explicitly instead of performing a couple of function calls
on each `Less` invocation.

```
name                             old time/op    new time/op    delta
IndexJoin/Cockroach-24             6.30ms ± 1%    5.78ms ± 1%  -8.31%  (p=0.000 n=10+10)
IndexJoin/MultinodeCockroach-24    8.01ms ± 1%    7.51ms ± 1%  -6.28%  (p=0.000 n=10+10)

name                             old alloc/op   new alloc/op   delta
IndexJoin/Cockroach-24             1.55MB ± 0%    1.57MB ± 0%  +0.98%  (p=0.000 n=9+10)
IndexJoin/MultinodeCockroach-24    2.28MB ± 2%    2.30MB ± 1%    ~     (p=0.400 n=10+9)

name                             old allocs/op  new allocs/op  delta
IndexJoin/Cockroach-24              8.16k ± 1%     8.13k ± 1%    ~     (p=0.160 n=10+10)
IndexJoin/MultinodeCockroach-24     12.7k ± 1%     12.6k ± 0%    ~     (p=0.128 n=10+9)
```
```
name                                                    old time/op    new time/op    delta
LookupJoinEqColsAreKeyNoOrdering/Cockroach-24             6.89ms ± 1%    6.43ms ± 1%  -6.65%  (p=0.000 n=10+10)
LookupJoinEqColsAreKeyNoOrdering/MultinodeCockroach-24    8.03ms ± 1%    7.48ms ± 2%  -6.92%  (p=0.000 n=10+10)
LookupJoinNoOrdering/Cockroach-24                         9.21ms ± 3%    8.82ms ± 5%  -4.23%  (p=0.007 n=10+10)
LookupJoinNoOrdering/MultinodeCockroach-24                11.9ms ± 3%    11.5ms ± 3%  -3.36%  (p=0.002 n=9+10)

name                                                    old alloc/op   new alloc/op   delta
LookupJoinEqColsAreKeyNoOrdering/Cockroach-24             1.81MB ± 1%    1.84MB ± 0%  +1.23%  (p=0.000 n=10+10)
LookupJoinEqColsAreKeyNoOrdering/MultinodeCockroach-24    2.50MB ± 2%    2.54MB ± 1%  +1.76%  (p=0.004 n=10+10)
LookupJoinNoOrdering/Cockroach-24                         1.89MB ± 0%    1.91MB ± 1%  +1.09%  (p=0.000 n=9+9)
LookupJoinNoOrdering/MultinodeCockroach-24                2.37MB ± 2%    2.42MB ± 4%  +1.85%  (p=0.010 n=10+9)

name                                                    old allocs/op  new allocs/op  delta
LookupJoinEqColsAreKeyNoOrdering/Cockroach-24              10.8k ± 0%     10.8k ± 1%    ~     (p=0.615 n=10+10)
LookupJoinEqColsAreKeyNoOrdering/MultinodeCockroach-24     15.1k ± 1%     15.0k ± 0%    ~     (p=0.101 n=10+10)
LookupJoinNoOrdering/Cockroach-24                          13.3k ± 1%     13.3k ± 1%    ~     (p=0.549 n=10+9)
LookupJoinNoOrdering/MultinodeCockroach-24                 17.3k ± 1%     17.3k ± 1%    ~     (p=0.460 n=10+8)
```

Addresses: #82159

Release note: None

82740: build: remove crdb-protobuf-client node_modules with ui-maintainer-clean r=maryliag,rickystewart a=sjbarag

Since version 33 [1], dev ui clean --all removes the
pkg/ui/workspaces/db-console/src/js/node_modules tree. Remove that tree
with make ui-maintainer-clean to keep parity between the two build
systems.

[1] 2e9e7a5 (dev: bump to version 33, 2022-05-27)

Release note: None

82744: ui: update cluster-ui to v22.2.0-prerelease-2 r=maryliag a=maryliag

Update cluster-ui to the latest value publishes

Release note: None

82748: ci: skip Docker test in CI r=ZhouXing19 a=rickystewart

This has been flaky for a while, skipping until we have more information
about what's going on here.

Release note: None

Co-authored-by: Alex Sarkesian <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
Co-authored-by: Sean Barag <[email protected]>
Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
  • Loading branch information
@AlexTalks @yuzefovich
@sjbarag @maryliag @rickystewart
6 people committed Jun 10, 2022
6 parents aac9c44 + 90eb477 + 76175be + e9ac4e6 + c869c9a + ad1550c commit f2cc0b3
Show file tree
Hide file tree
Showing 10 changed files with 574 additions and 82 deletions.
2 changes: 1 addition & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1453,7 +1453,7 @@ ui-clean: ## Remove build artifacts.
ui-maintainer-clean: ## Like clean, but also remove installed dependencies
ui-maintainer-clean: ui-clean
$(info $(yellow)NOTE: consider using `./dev ui clean --all` instead of `make ui-maintainer-clean`$(term-reset))
rm -rf pkg/ui/node_modules pkg/ui/workspaces/db-console/node_modules pkg/ui/yarn.installed pkg/ui/workspaces/cluster-ui/node_modules
rm -rf pkg/ui/node_modules pkg/ui/workspaces/db-console/node_modules pkg/ui/yarn.installed pkg/ui/workspaces/cluster-ui/node_modules pkg/ui/workspaces/db-console/src/js/node_modules

pkg/roachprod/vm/aws/embedded.go: bin/.bootstrap pkg/roachprod/vm/aws/config.json pkg/roachprod/vm/aws/old.json bin/terraformgen
(cd pkg/roachprod/vm/aws && $(GO) generate)
Expand Down
9 changes: 5 additions & 4 deletions build/teamcity/cockroach/ci/tests/docker_image.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ tc_prepare

tc_start_block "Run docker image tests"

bazel run \
//pkg/testutils/docker:docker_test \
--config=crosslinux --config=test \
--test_timeout=3000
# Skip for now: #82747
#bazel run \
# //pkg/testutils/docker:docker_test \
# --config=crosslinux --config=test \
# --test_timeout=3000

tc_end_block "Run docker image tests"
69 changes: 69 additions & 0 deletions pkg/bench/bench_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1200,6 +1200,75 @@ func BenchmarkIndexJoinColumnFamilies(b *testing.B) {
})
}

// BenchmarkLookupJoinEqColsAreKeyNoOrdering measures a lookup-join with 1000
// rows when equality columns are key and ordering doesn't have to be
// maintained.
func BenchmarkLookupJoinEqColsAreKeyNoOrdering(b *testing.B) {
defer log.Scope(b).Close(b)
ForEachDB(b, func(b *testing.B, db *sqlutils.SQLRunner) {
db.Exec(b, `CREATE TABLE t1 (a INT)`)
db.Exec(b, `INSERT INTO t1 SELECT generate_series(1, 1000)`)
db.Exec(b, `CREATE TABLE t2 (a INT PRIMARY KEY, b INT)`)
db.Exec(b, `INSERT INTO t2 SELECT generate_series(1, 1000), (random()*1000)::INT`)
b.ResetTimer()

for i := 0; i < b.N; i++ {
db.Exec(b, `SELECT * FROM t1 INNER LOOKUP JOIN t2 ON t1.a = t2.a`)
}
})
}

// BenchmarkLookupJoinEqColsAreKeyOrdering measures a lookup-join with 1000 rows
// when equality columns are key and ordering needs to be maintained.
func BenchmarkLookupJoinEqColsAreKeyOrdering(b *testing.B) {
defer log.Scope(b).Close(b)
ForEachDB(b, func(b *testing.B, db *sqlutils.SQLRunner) {
db.Exec(b, `CREATE TABLE t1 (a INT PRIMARY KEY)`)
db.Exec(b, `INSERT INTO t1 SELECT generate_series(1, 1000)`)
db.Exec(b, `CREATE TABLE t2 (a INT PRIMARY KEY, b INT)`)
db.Exec(b, `INSERT INTO t2 SELECT generate_series(1, 1000), (random()*1000)::INT`)
b.ResetTimer()

for i := 0; i < b.N; i++ {
db.Exec(b, `SELECT * FROM t1 INNER LOOKUP JOIN t2 ON t1.a = t2.a ORDER BY t1.a`)
}
})
}

// BenchmarkLookupJoinNoOrdering measures a lookup-join with 1000 rows and
// ordering doesn't have to be maintained.
func BenchmarkLookupJoinNoOrdering(b *testing.B) {
defer log.Scope(b).Close(b)
ForEachDB(b, func(b *testing.B, db *sqlutils.SQLRunner) {
db.Exec(b, `CREATE TABLE t1 (a INT)`)
db.Exec(b, `INSERT INTO t1 SELECT generate_series(1, 1000)`)
db.Exec(b, `CREATE TABLE t2 (a INT, INDEX (a))`)
db.Exec(b, `INSERT INTO t2 SELECT generate_series(1, 1000)`)
b.ResetTimer()

for i := 0; i < b.N; i++ {
db.Exec(b, `SELECT * FROM t1 INNER LOOKUP JOIN t2 ON t1.a = t2.a`)
}
})
}

// BenchmarkLookupJoinOrdering measures a lookup-join with 1000 rows when
// ordering needs to be maintained.
func BenchmarkLookupJoinOrdering(b *testing.B) {
defer log.Scope(b).Close(b)
ForEachDB(b, func(b *testing.B, db *sqlutils.SQLRunner) {
db.Exec(b, `CREATE TABLE t1 (a INT PRIMARY KEY)`)
db.Exec(b, `INSERT INTO t1 SELECT generate_series(1, 1000)`)
db.Exec(b, `CREATE TABLE t2 (a INT, INDEX (a))`)
db.Exec(b, `INSERT INTO t2 SELECT generate_series(1, 1000)`)
b.ResetTimer()

for i := 0; i < b.N; i++ {
db.Exec(b, `SELECT * FROM t1 INNER LOOKUP JOIN t2 ON t1.a = t2.a ORDER BY t1.a`)
}
})
}

func BenchmarkSortJoinAggregation(b *testing.B) {
defer log.Scope(b).Close(b)
ForEachDB(b, func(b *testing.B, db *sqlutils.SQLRunner) {
Expand Down
1 change: 1 addition & 0 deletions pkg/cmd/roachtest/tests/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ go_library(
"costfuzz.go",
"decommission.go",
"decommission_self.go",
"decommissionbench.go",
"disk_full.go",
"disk_stall.go",
"django.go",
Expand Down
175 changes: 102 additions & 73 deletions pkg/cmd/roachtest/tests/decommission.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,63 +245,9 @@ func runDecommission(
}
c.Run(ctx, c.Node(pinnedNode), `./workload init kv --drop`)

waitReplicatedAwayFrom := func(downNodeID int) error {
db := c.Conn(ctx, t.L(), pinnedNode)
defer func() {
_ = db.Close()
}()

for {
var count int
if err := db.QueryRow(
// Check if the down node has any replicas.
"SELECT count(*) FROM crdb_internal.ranges WHERE array_position(replicas, $1) IS NOT NULL",
downNodeID,
).Scan(&count); err != nil {
return err
}
if count == 0 {
fullReplicated := false
if err := db.QueryRow(
// Check if all ranges are fully replicated.
"SELECT min(array_length(replicas, 1)) >= 3 FROM crdb_internal.ranges",
).Scan(&fullReplicated); err != nil {
return err
}
if fullReplicated {
break
}
}
time.Sleep(time.Second)
}
return nil
}

waitUpReplicated := func(targetNode, targetNodeID int) error {
db := c.Conn(ctx, t.L(), pinnedNode)
defer func() {
_ = db.Close()
}()

var count int
for {
// Check to see that there are no ranges where the target node is
// not part of the replica set.
stmtReplicaCount := fmt.Sprintf(
`SELECT count(*) FROM crdb_internal.ranges WHERE array_position(replicas, %d) IS NULL and database_name = 'kv';`, targetNodeID)
if err := db.QueryRow(stmtReplicaCount).Scan(&count); err != nil {
return err
}
t.Status(fmt.Sprintf("node%d missing %d replica(s)", targetNode, count))
if count == 0 {
break
}
time.Sleep(time.Second)
}
return nil
}
h := newDecommTestHelper(t, c)

if err := waitReplicatedAwayFrom(0 /* no down node */); err != nil {
if err := h.waitReplicatedAwayFrom(ctx, 0 /* no down node */, pinnedNode); err != nil {
t.Fatal(err)
}

Expand Down Expand Up @@ -345,18 +291,6 @@ func runDecommission(
return nodeID, nil
}

stop := func(node int) error {
port := fmt.Sprintf("{pgport:%d}", node)
defer time.Sleep(time.Second) // work around quit returning too early
return c.RunE(ctx, c.Node(node), "./cockroach quit --insecure --host=:"+port)
}

decom := func(id int) error {
port := fmt.Sprintf("{pgport:%d}", pinnedNode) // always use the pinned node
t.Status(fmt.Sprintf("decommissioning node %d", id))
return c.RunE(ctx, c.Node(pinnedNode), fmt.Sprintf("./cockroach node decommission --insecure --wait=all --host=:%s %d", port, id))
}

tBegin, whileDown := timeutil.Now(), true
node := nodes
for timeutil.Since(tBegin) <= duration {
Expand All @@ -376,28 +310,29 @@ func runDecommission(
}

run(fmt.Sprintf(`ALTER RANGE default CONFIGURE ZONE = 'constraints: {"+node%d"}'`, node))
if err := waitUpReplicated(node, nodeID); err != nil {
if err := h.waitUpReplicated(ctx, nodeID, node, "kv"); err != nil {
return err
}

if whileDown {
if err := stop(node); err != nil {
if err := h.stop(ctx, node); err != nil {
return err
}
}

run(fmt.Sprintf(`ALTER RANGE default CONFIGURE ZONE = 'constraints: {"-node%d"}'`, node))

if err := decom(nodeID); err != nil {
targetNodeList := option.NodeListOption{nodeID}
if _, err := h.decommission(ctx, targetNodeList, pinnedNode, "--wait=all"); err != nil {
return err
}

if err := waitReplicatedAwayFrom(nodeID); err != nil {
if err := h.waitReplicatedAwayFrom(ctx, nodeID, pinnedNode); err != nil {
return err
}

if !whileDown {
if err := stop(node); err != nil {
if err := h.stop(ctx, node); err != nil {
return err
}
}
Expand Down Expand Up @@ -1230,6 +1165,13 @@ func newDecommTestHelper(t test.Test, c cluster.Cluster) *decommTestHelper {
}
}

// stop gracefully stops a node with `cockroach quit`.
func (h *decommTestHelper) stop(ctx context.Context, node int) error {
port := fmt.Sprintf("{pgport:%d}", node)
defer time.Sleep(time.Second) // work around quit returning too early
return h.c.RunE(ctx, h.c.Node(node), "./cockroach quit --insecure --host=:"+port)
}

// decommission decommissions the given targetNodes, running the process
// through the specified runNode.
func (h *decommTestHelper) decommission(
Expand Down Expand Up @@ -1266,6 +1208,93 @@ func (h *decommTestHelper) recommission(
return execCLI(ctx, h.t, h.c, runNode, args...)
}

// checkDecommissioned validates that a node has successfully decommissioned
// and has updated its state in gossip.
func (h *decommTestHelper) checkDecommissioned(ctx context.Context, downNodeID, runNode int) error {
db := h.c.Conn(ctx, h.t.L(), runNode)
defer db.Close()
var membership string
if err := db.QueryRow("SELECT membership FROM crdb_internal.kv_node_liveness WHERE node_id = $1",
downNodeID).Scan(&membership); err != nil {
return err
}

if membership != "decommissioned" {
return errors.Newf("node %d not decommissioned", downNodeID)
}

return nil
}

// waitReplicatedAwayFrom checks each second until there are no ranges present
// on a node and all ranges are fully replicated.
func (h *decommTestHelper) waitReplicatedAwayFrom(
ctx context.Context, downNodeID, runNode int,
) error {
db := h.c.Conn(ctx, h.t.L(), runNode)
defer func() {
_ = db.Close()
}()

for {
var count int
if err := db.QueryRow(
// Check if the down node has any replicas.
"SELECT count(*) FROM crdb_internal.ranges WHERE array_position(replicas, $1) IS NOT NULL",
downNodeID,
).Scan(&count); err != nil {
return err
}
if count == 0 {
fullReplicated := false
if err := db.QueryRow(
// Check if all ranges are fully replicated.
"SELECT min(array_length(replicas, 1)) >= 3 FROM crdb_internal.ranges",
).Scan(&fullReplicated); err != nil {
return err
}
if fullReplicated {
break
}
}
time.Sleep(time.Second)
}
return nil
}

// waitUpReplicated checks each second until all ranges in a particular
// database are replicated on a particular node, with targetNode representing
// the roachprod node and targetNodeID representing the logical nodeID within
// the cluster.
func (h *decommTestHelper) waitUpReplicated(
ctx context.Context, targetNodeID, targetNode int, database string,
) error {
db := h.c.Conn(ctx, h.t.L(), targetNode)
defer func() {
_ = db.Close()
}()

var count int
for {
// Check to see that there are no ranges where the target node is
// not part of the replica set.
stmtReplicaCount := fmt.Sprintf(
"SELECT count(*) FROM crdb_internal.ranges "+
"WHERE array_position(replicas, %d) IS NULL and database_name = '%s';",
targetNodeID, database,
)
if err := db.QueryRow(stmtReplicaCount).Scan(&count); err != nil {
return err
}
h.t.L().Printf("node%d (n%d) awaiting %d replica(s)", targetNode, targetNodeID, count)
if count == 0 {
break
}
time.Sleep(time.Second)
}
return nil
}

// expectColumn constructs a matching regex for a given column (identified
// by its column index).
func (h *decommTestHelper) expectColumn(
Expand Down
Loading

0 comments on commit f2cc0b3

Please sign in to comment.