Skip to content

Commit

Permalink
changefeedccl: Bump default per changefeed limit
Browse files Browse the repository at this point in the history
Inrease default changefeed limit to 512MiB.
The limit was previously reduced from 1GiB due to the
concerns of changefeed impact on foreground latency.
That impact in turn was due how changefeed allocated and
management memory.  The issue leading to the reduction
in this setting default have been addressed (via
improvememnts in memory management, leading to the
changefeed impact reduction on the Go runtime, as well as
addition of CPU pacing).

The 128MiB default limit is too low for file based sinks.
By default, those sinks buffer 16MB of data, and often
times this is combined with compression.  If the data
is highly compressible (5-10x, which is not uncommon),
changefeds wind up using the entirety of memory budget,
before reaching their target file size, resulting in small
files being produced.

This PR splits a difference between original, 1GiB setting,
and the new 128MiB setting by increasing default setting to 1/2GiB.

Epic: None

Release note (enterprise change): Bump up default
`changefeed.memory.per_changefeed_limit` setting to 1/2GiB.
This should result in changefeeds being able to produce larger
files.
  • Loading branch information
Yevgeniy Miretskiy committed Feb 1, 2023
1 parent 8ae413b commit 54dd78c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pkg/ccl/changefeedccl/changefeedbase/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ var PerChangefeedMemLimit = settings.RegisterByteSizeSetting(
settings.TenantWritable,
"changefeed.memory.per_changefeed_limit",
"controls amount of data that can be buffered per changefeed",
1<<27, // 128MiB
1<<29, // 512MiB
)

// SlowSpanLogThreshold controls when we will log slow spans.
Expand Down

0 comments on commit 54dd78c

Please sign in to comment.