Skip to content

Adding task pending time in MSQ reports#15966

Merged
cryptoe merged 4 commits intoapache:masterfrom
cryptoe:task_pending_time_msq
Feb 27, 2024
Merged

Adding task pending time in MSQ reports#15966
cryptoe merged 4 commits intoapache:masterfrom
cryptoe:task_pending_time_msq

Conversation

@cryptoe
Copy link
Copy Markdown
Contributor

@cryptoe cryptoe commented Feb 26, 2024

  • Added a new field pendingMs in MSQ task reports. This helps in figuring out the exact run time of the MSQ worker tasks.
  • Fixed data races.

The payload now looks like

"workers": {
          "0": [
            {
              "workerId": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e-worker0_0",
              "state": "SUCCESS",
              "durationMs": 15511,
              "pendingMs": 137
            }
          ]
        }

Release notes

A new field in MSQ task report which captures the milliseconds elapsed between when the worker task was first requested and when it fully started RUNNING. Actual work time can be calculated using actualWorkTimeMS = durationMs - pendingMs

@github-actions github-actions Bot added Area - Documentation Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Feb 26, 2024
}

@JsonProperty("pendingMs")
public long getPendingTimeInMS()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pendingTimeMillis would probably be a better name and more in line with rest of the names in Druid.

Copy link
Copy Markdown
Contributor Author

@cryptoe cryptoe Feb 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we already have durationMs as part of line:389 of this payload, having pendingTimeMillis would be weird. If you feel strongly about it, I can change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using millis as convention in this class, so it's a bit unfortunate that we already have "durationMs" in the taskReport. Would the best thing be to call it pendingTimeMillis in the code and expose it in the report as getPendingTimeInMs. As a nit, I think it should be getPendingTimeInMs though.

Copy link
Copy Markdown
Contributor

@adarshsanjeev adarshsanjeev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, minor comments

final TaskTracker tracker = taskEntry.getValue();
if (!canceledWorkerTasks.contains(taskId)
&& (tracker.status == null || !tracker.status.getStatusCode().isComplete())) {
if ((!canceledWorkerTasks.contains(taskId))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this looks a bit weird due to the parenthesis and new lines.

Suggested change
if ((!canceledWorkerTasks.contains(taskId))
if (!canceledWorkerTasks.contains(taskId) && !tracker.isComplete()) {

Copy link
Copy Markdown
Contributor Author

@cryptoe cryptoe Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally find 2 not statements a little tricky so hence like the parenthesis. Keeping it as is for now.

@cryptoe cryptoe merged commit 5bb5b41 into apache:master Feb 27, 2024
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - Documentation Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants