[SPARK-16166][Core] Also take off-heap memory usage into consideration in log and webui display

[jerryshao]

What changes were proposed in this pull request?

Currently in the log and UI display, only on-heap storage memory is calculated and displayed,

16/06/27 13:41:52 INFO MemoryStore: Block rdd_5_0 stored as values in memory (estimated size 17.8 KB, free 665.9 MB)

With SPARK-13992 off-heap memory is supported for data persistence, so here change to also take off-heap storage memory into consideration.

How was this patch tested?

Unit test and local verification.

[SparkQA]

Test build #61291 has finished for PR 13920 at commit e3194bd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

@andrewor14 @JoshRosen would you please help to review this patch, thanks a lot.

[JoshRosen]

Sorry for the delay in reviewing; I've been super busy and stuff has slipped through the cracks. LGTM because this looks like a net improvement over what we have today. In the longer term I think that it might be slightly confusing to combine both on- and off-heap storage memory into a single total in the UI, but we can always deal with that later if it turns out to be a problem.

Let me quickly re-test this to make sure that it still compiles and passes tests, then I'll merge it if Jenkins approves.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #62837 has finished for PR 13920 at commit e3194bd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Thanks a lot @JoshRosen .