[SPARK-14628][CORE][folllow-up] Always tracking read/write metrics #12462
Conversation
|
cc @rxin |
| "attemptId" : "1", | ||
| "startTimeEpoch" : 1430917380880, | ||
| "endTimeEpoch" : 1430917380890, | ||
| "lastUpdatedEpoch" : 0, |
There was a problem hiding this comment.
When I re-generate the gold answer, these 3 fields are moved to the last. It seems that when we introduce these 3 fields at #11326, we didn't re-generate gold answer, but update the answer files manually.
|
LGTM pending Jenkins. |
|
Test build #2803 has finished for PR 12462 at commit
|
| val outputMetrics: Option[OutputMetrics], | ||
| val shuffleReadMetrics: Option[ShuffleReadMetrics], | ||
| val shuffleWriteMetrics: Option[ShuffleWriteMetrics]) | ||
| val inputMetrics: InputMetrics, |
There was a problem hiding this comment.
one thing - can you verify that this doesn't change the json output?
There was a problem hiding this comment.
I couldn't find the jackson doc for it, so I tried it locally. For scala Option, jackson just treat it as a nullable field, if it's Some, it will unwrap the Some and parse the real value. If it's None, it will return null for that field.
In this case, where we always return Some, it's ok to drop the Option.
|
retest this please |
|
Test build #2811 has started for PR 12462 at commit |
|
Test build #56072 has finished for PR 12462 at commit
|
|
Test build #2815 has finished for PR 12462 at commit
|
|
Merging in master. |
## What changes were proposed in this pull request? This PR is a follow up for apache#12417, now we always track input/output/shuffle metrics in spark JSON protocol and status API. Most of the line changes are because of re-generating the gold answer for `HistoryServerSuite`, and we add a lot of 0 values for read/write metrics. ## How was this patch tested? existing tests. Author: Wenchen Fan <wenchen@databricks.com> Closes apache#12462 from cloud-fan/follow.
3 participants
What changes were proposed in this pull request?
This PR is a follow up for #12417, now we always track input/output/shuffle metrics in spark JSON protocol and status API.
Most of the line changes are because of re-generating the gold answer for
HistoryServerSuite, and we add a lot of 0 values for read/write metrics.How was this patch tested?
existing tests.