Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding task pending time in MSQ reports #15966

Merged
merged 4 commits into from
Feb 27, 2024

Conversation

cryptoe
Copy link
Contributor

@cryptoe cryptoe commented Feb 26, 2024

  • Added a new field pendingMs in MSQ task reports. This helps in figuring out the exact run time of the MSQ worker tasks.
  • Fixed data races.

The payload now looks like

"workers": {
          "0": [
            {
              "workerId": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e-worker0_0",
              "state": "SUCCESS",
              "durationMs": 15511,
              "pendingMs": 137
            }
          ]
        }

Release notes

A new field in MSQ task report which captures the milliseconds elapsed between when the worker task was first requested and when it fully started RUNNING. Actual work time can be calculated using actualWorkTimeMS = durationMs - pendingMs

@github-actions github-actions bot added Area - Documentation Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Feb 26, 2024
@@ -390,6 +391,12 @@ public long getDuration()
{
return duration;
}

@JsonProperty("pendingMs")
public long getPendingTimeInMS()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pendingTimeMillis would probably be a better name and more in line with rest of the names in Druid.

Copy link
Contributor Author

@cryptoe cryptoe Feb 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we already have durationMs as part of line:389 of this payload, having pendingTimeMillis would be weird. If you feel strongly about it, I can change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using millis as convention in this class, so it's a bit unfortunate that we already have "durationMs" in the taskReport. Would the best thing be to call it pendingTimeMillis in the code and expose it in the report as getPendingTimeInMs. As a nit, I think it should be getPendingTimeInMs though.

Copy link
Contributor

@adarshsanjeev adarshsanjeev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, minor comments

@@ -737,8 +750,9 @@ private void shutDownTasks()
for (final Map.Entry<String, TaskTracker> taskEntry : taskTrackers.entrySet()) {
final String taskId = taskEntry.getKey();
final TaskTracker tracker = taskEntry.getValue();
if (!canceledWorkerTasks.contains(taskId)
&& (tracker.status == null || !tracker.status.getStatusCode().isComplete())) {
if ((!canceledWorkerTasks.contains(taskId))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this looks a bit weird due to the parenthesis and new lines.

Suggested change
if ((!canceledWorkerTasks.contains(taskId))
if (!canceledWorkerTasks.contains(taskId) && !tracker.isComplete()) {

Copy link
Contributor Author

@cryptoe cryptoe Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally find 2 not statements a little tricky so hence like the parenthesis. Keeping it as is for now.

@cryptoe cryptoe merged commit 5bb5b41 into apache:master Feb 27, 2024
82 of 83 checks passed
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area - Batch Ingestion Area - Documentation Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants