Update default value of druid.indexer.tasklock.batchAllocationWaitTime to zero
#16578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kfaraz
commented
Jun 10, 2024
kfaraz
changed the title
druid.indexer.tasklock.batchAllocationWaitTime to zerodruid.indexer.tasklock.batchAllocationWaitTime to zero
LakshSingla
approved these changes
Jun 10, 2024
| @@ -1120,7 +1120,7 @@ These Overlord static configurations can be defined in the `overlord/runtime.pro | |||
| |`druid.indexer.storage.recentlyFinishedThreshold`|Duration of time to store task results. Default is 24 hours. If you have hundreds of tasks running in a day, consider increasing this threshold.|`PT24H`| | |||
| |`druid.indexer.tasklock.forceTimeChunkLock`|_**Setting this to false is still experimental**_<br/> If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/tasks.md#context). See [Task Locking & Priority](../ingestion/tasks.md#context) for more details about locking in tasks.|true| | |||
| |`druid.indexer.tasklock.batchSegmentAllocation`| If set to true, Druid performs segment allocate actions in batches to improve throughput and reduce the average `task/action/run/time`. See [batching `segmentAllocate` actions](../ingestion/tasks.md#batching-segmentallocate-actions) for details.|true| | |||
| |`druid.indexer.tasklock.batchAllocationWaitTime`|Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if `batchSegmentAllocation` is enabled.|500| | |||
| |`druid.indexer.tasklock.batchAllocationWaitTime`|Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if `batchSegmentAllocation` is enabled.|0| | |||
There was a problem hiding this comment.
Should we specify the behavior when this value is set to 0, as done in the description of the PR?
There was a problem hiding this comment.
I feel it is self-explanatory, a value of zero means no waiting unless something else is already being processed. If you still feel it warrants clarification, I can add a line to this effect.
|
Thanks for the review, @LakshSingla ! I have updated the release note. Please let me know if it seems adequate. |
|
Thanks for the changes! |
|
Thanks for the change @kfaraz |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The current default
batchAllocationWaitTimeis 500ms. This means that theSegmentAllocationQueuewaits for 500ms before a batch is processed to allow it to accumulate more requests.
In some cases, particularly MSQ tasks, this adds an unnecessary penalty to the task duration.
Changes
With batch segment allocation enabled and
batchAllocationWaitTime=0,the segment allocation queue becomes more adaptive and exhibits the following behaviour:
Testing
We have had
batchAllocationWaitTime=0in several of our production clusters for a couple of months now.It has particularly helped with MSQ tasks, which don't really benefit from batching as segments are allocated serially.
Waiting 500ms for every allocation increased MSQ task durations leading to unnecessary compute costs.
Release Note
Update default value of
druid.indexer.tasklock.batchAllocationWaitTimeto0.Thus, a segment allocation request is processed immediately unless there are already some requests queued before this one. While in queue, a segment allocation request may get clubbed together with other similar requests into a batch to reduce load on the metadata store.