Skip to content

Commit

Permalink
changefeedccl: use numcpu >> 2 workers for event consumers
Browse files Browse the repository at this point in the history
Previously, a default value of 8 was used for the kvevent parallel consumer.
The reason for this was that we observed performance improvements in a 15 node
32 VCPU cluster when we increased this parameter to 8. After 8, the
improvements were much smaller.

The issue with a default of 8 is that that on smaller machines, 8
workers can be too much overhead, especially since the work is CPU intensive.

This change updates the default to be runtime.NumCPU() >> 2 workers, which
aligns with using 8 workers on 32 VCPU machines.

Fixes cockroachdb#89589
Epic: none

Release note: None
  • Loading branch information
jayshrivastava committed Oct 10, 2022
1 parent 9e7e704 commit 69a1047
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion pkg/ccl/changefeedccl/changefeedbase/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ package changefeedbase

import (
"encoding/json"
"runtime"
"time"

"github.com/cockroachdb/cockroach/pkg/settings"
Expand Down Expand Up @@ -239,7 +240,13 @@ var EventConsumerWorkers = settings.RegisterIntSetting(
settings.TenantWritable,
"changefeed.event_consumer_workers",
"the number of workers to use when processing events; 0 or 1 disables",
int64(util.ConstantWithMetamorphicTestRange("changefeed.consumer_max_workers", 8, 0, 32)),
func() int64 {
idealNumber := runtime.NumCPU() >> 2
if idealNumber < 1 {
return 1
}
return int64(idealNumber)
}(),
settings.NonNegativeInt,
).WithPublic()

Expand Down

0 comments on commit 69a1047

Please sign in to comment.