docs: msq autocompaction #16681
docs: msq autocompaction #16681
Conversation
Co-authored-by: Kashif Faraz <[email protected]> Co-authored-by: Vishesh Garg <[email protected]>
| * Have the [MSQ task engine extension loaded](../multi-stage-query/index.md#load-the-extension). | ||
| * In your Overlord runtime properties, set the following properties: | ||
| * `druid.supervisor.compaction.enabled` to `true` so that compaction tasks can be run as a supervisor task | ||
| * `druid.supervisor.compaction.engine` to `msq` to specify the MSQ task engine as the compaction engine |
There was a problem hiding this comment.
druid.supervisor.compaction.engine is only for setting the default engine -- which is native otherwise --when no engine is explicitly specified in the spec. This point can be mentioned separately.
So it could be put as:
Either set spec.engine to msq in the supervisor spec, or omit spec.engine in the supervisor spec and set druid.supervisor.compaction.engine runtime property on the overlord to msq
| "dataSource": "wikipedia", // required | ||
| "tuningConfig": {...}, // optional | ||
| "granularitySpec": {...}, // optional | ||
| ... |
There was a problem hiding this comment.
We should also add engine parameter here
"granularitySpec": {...},
"engine": <native|msq>, // optional
...
There was a problem hiding this comment.
I understand that skipping it here and specifying it later in supervisor-based spec may be confusing. If we keep it here, just want to make sure that users realize that it's only supported with supervisors.
Also, we need to add this field to Automatic compaction dynamic configuration page. Maybe this info can reside there simiar to below:
engine | Engine for compaction. Can be either native or msq. MSQ is only supported with compaction supervisors | no (default = native)
There was a problem hiding this comment.
Shall we also include the above on Automatic compaction dynamic configuration page?
Co-authored-by: Vishesh Garg <[email protected]>
| * Have the [MSQ task engine extension loaded](../multi-stage-query/index.md#load-the-extension). | ||
| * In your Overlord runtime properties, set the following properties: | ||
| * `druid.supervisor.compaction.enabled` to `true` so that compaction tasks can be run as a supervisor task | ||
| * Optionally, set `druid.supervisor.compaction.engine` to `msq` to specify the MSQ task engine as the default compaction engine. If you don't do this, you'll need to set `spec.engine` to `msq` for each compaction supervisor spec where you want to use the MSQ task engine. |
There was a problem hiding this comment.
Setting this also in Overlord?
There was a problem hiding this comment.
Yes, to nudge people towards using compaction supervisors, the msq engine is now only supported with supervisor-based compaction on the overlord -- not on the coordinator. So above properties are set on the overlord.
| "dataSource": "wikipedia", // required | ||
| "tuningConfig": {...}, // optional | ||
| "granularitySpec": {...}, // optional | ||
| ... |
There was a problem hiding this comment.
Shall we also include the above on Automatic compaction dynamic configuration page?
| - Only dynamic and range-based partitioning are supported | ||
| - Set `rollup` to `true` if and only if `metricSpec` is not empty or null. | ||
| - You can only partition on string dimensions. However, multi-valued string dimensions are not supported. | ||
| - The `maxTotalRows` config is not supported in `DynamicPartitionsSpec`. Use `maxRowsPerSegment` instead. |
There was a problem hiding this comment.
Same as #16681 (comment)
Co-authored-by: Vishesh Garg <[email protected]>
Co-authored-by: Vishesh Garg <[email protected]>
Co-authored-by: Victoria Lim <[email protected]>
Co-authored-by: Victoria Lim <[email protected]>
Co-authored-by: Victoria Lim <[email protected]>
Co-authored-by: Kashif Faraz <[email protected]> Co-authored-by: Vishesh Garg <[email protected]> Co-authored-by: Victoria Lim <[email protected]> (cherry picked from commit d1b81f3)
Co-authored-by: Kashif Faraz <[email protected]> Co-authored-by: Vishesh Garg <[email protected]> Co-authored-by: Victoria Lim <[email protected]>
Working with @gargvishesh to create the docs for #16291
These docs are written for being able to set the compaction engine at the datasource level. They'll be updated (in this PR or a followup depending on timing) when the clusterwide setting is available
Release note
n/a. Will be in the dev PR.
Key changed/added classes in this PR
MyFooOurBarTheirBazThis PR has: