Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

receive: Expose write chunk queue as flag #5566

Merged
merged 2 commits into from
Aug 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#5527](https://github.com/thanos-io/thanos/pull/5527) Receive: Add per request limits for remote write.
- [#5520](https://github.com/thanos-io/thanos/pull/5520) Receive: Meta-monitoring based active series limiting
- [#5555](https://github.com/thanos-io/thanos/pull/5555) Query: Added `--query.active-query-path` flag, allowing the user to configure the directory to create an active query tracking file, `queries.active`, for different resolution.
- [#5566](https://github.com/thanos-io/thanos/pull/5566) Receive: Added experimental support to enable chunk write queue via `--tsdb.write-queue-size` flag.

### Changed

Expand Down
23 changes: 15 additions & 8 deletions cmd/thanos/receive.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,15 @@ func registerReceive(app *extkingpin.App) {
}

tsdbOpts := &tsdb.Options{
MinBlockDuration: int64(time.Duration(*conf.tsdbMinBlockDuration) / time.Millisecond),
MaxBlockDuration: int64(time.Duration(*conf.tsdbMaxBlockDuration) / time.Millisecond),
RetentionDuration: int64(time.Duration(*conf.retention) / time.Millisecond),
NoLockfile: conf.noLockFile,
WALCompression: conf.walCompression,
AllowOverlappingBlocks: conf.tsdbAllowOverlappingBlocks,
MaxExemplars: conf.tsdbMaxExemplars,
EnableExemplarStorage: true,
MinBlockDuration: int64(time.Duration(*conf.tsdbMinBlockDuration) / time.Millisecond),
MaxBlockDuration: int64(time.Duration(*conf.tsdbMaxBlockDuration) / time.Millisecond),
RetentionDuration: int64(time.Duration(*conf.retention) / time.Millisecond),
NoLockfile: conf.noLockFile,
WALCompression: conf.walCompression,
AllowOverlappingBlocks: conf.tsdbAllowOverlappingBlocks,
MaxExemplars: conf.tsdbMaxExemplars,
EnableExemplarStorage: true,
HeadChunksWriteQueueSize: int(conf.tsdbWriteQueueSize),
}

// Are we running in IngestorOnly, RouterOnly or RouterIngestor mode?
Expand Down Expand Up @@ -787,6 +788,7 @@ type receiveConfig struct {
tsdbMaxBlockDuration *model.Duration
tsdbAllowOverlappingBlocks bool
tsdbMaxExemplars int64
tsdbWriteQueueSize int64

walCompression bool
noLockFile bool
Expand Down Expand Up @@ -889,6 +891,11 @@ func (rc *receiveConfig) registerFlag(cmd extkingpin.FlagClause) {
" ingesting a new exemplar will evict the oldest exemplar from storage. 0 (or less) value of this flag disables exemplars storage.").
Default("0").Int64Var(&rc.tsdbMaxExemplars)

cmd.Flag("tsdb.write-queue-size",
"[EXPERIMENTAL] Enables configuring the size of the chunk write queue used in the head chunks mapper. "+
"A queue size of zero (default) disables this feature entirely.").
Default("0").Hidden().Int64Var(&rc.tsdbWriteQueueSize)

cmd.Flag("hash-func", "Specify which hash function to use when calculating the hashes of produced files. If no function has been specified, it does not happen. This permits avoiding downloading some files twice albeit at some performance cost. Possible values are: \"\", \"SHA256\".").
Default("").EnumVar(&rc.hashFunc, "SHA256", "")

Expand Down