[SPARK-48788][CORE][UI] Expose task peak onheap/offheap execution memory to API and Spark UI#47872
[SPARK-48788][CORE][UI] Expose task peak onheap/offheap execution memory to API and Spark UI#47872liuzqt wants to merge 3 commits intoapache:masterfrom
Conversation
liuzqt
changed the title
|
Hi @mridulm @dongjoon-hyun I'm working on this follow-up item for #47776, mostly APIs change and Spark UI change, I've manually verified it in Spark UI, could you pls help review this when you have time? And BTW do you know how to run |
liuzqt
changed the title
liuzqt
changed the title
mridulm
left a comment
mridulm
left a comment
There was a problem hiding this comment.
Took a quick look, but I am not very familiar with this side of spark.
+CC @yaooqinn, @yanboliang as well for review.
| int64 shuffle_remote_reqs_duration = 50; | ||
| int64 shuffle_merged_remote_req_duration = 51; | ||
| int64 peak_on_heap_execution_memory = 52; | ||
| int64 peak_off_heap_execution_memory = 53; |
There was a problem hiding this comment.
Add to ExecutorStageSummary, ExecutorSummary, ExecutorMetricsDistributions as well.
(Here and other model classes)
There was a problem hiding this comment.
Looks like we already have executor level stage memory metrics:
ExecutorStageSummaryhasval peakMemoryMetrics: Option[ExecutorMetrics]ExecutorSummaryhasval memoryMetrics: Option[MemoryMetrics]ExecutorMetricsDistributionshasval peakMemoryMetrics: ExecutorPeakMetricsDistributions
which are aggregated through AppStatusListener.updateStageLevelPeakExecutorMetrics
|
Also +CC @gengliangwang |
|
Hi @mridulm I've fixed broken tests, also answered you questions, do you mind taking another look when you have time? Thanks |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
2 participants
What changes were proposed in this pull request?
#47776 has introduced task peak on/off heap execution memory metrics, this PR exposes these two metrics to APIs, also shown in Spark UI Stage page, specifically, in 3 relevant sections(see screen shot below):
Why are the changes needed?
Does this PR introduce any user-facing change?
Yes. Expose metrics to APIs, also show in Spark UI Stage page.
How was this patch tested?
Existing UTs.
Manually verify through Spark UI:
- html page
Spark shell - Details for Stage 4 (Attempt 0).mhtml.zip
- Stage details
- Task metrics summary
- Task metrics
Was this patch authored or co-authored using generative AI tooling?
NO