Skip to content

[GLUTEN-2139][CORE] Add gluten event and UI#2247

Merged
zhouyuan merged 2 commits intoapache:mainfrom
ulysses-you:fallback-event
Jul 10, 2023
Merged

[GLUTEN-2139][CORE] Add gluten event and UI#2247
zhouyuan merged 2 commits intoapache:mainfrom
ulysses-you:fallback-event

Conversation

@ulysses-you
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This pr adds a new module gluten-ui to holds the event and UI related code. So it would be more easy to put a simple gluten-ui jar instead of gluten-core jar into history server.

Adds two Gluten events:

  • GlutenBuildInfoEvent, used to show build information in UI
  • GlutenPlanFallbackEvent, used to show query execution fallback reason in UI

Introduce a new config spark.gluten.ui.enabled with enabled by default.

(Fixes: #2139)

How was this patch tested?

Add test and test locally

image

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jul 7, 2023

#2139

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jul 7, 2023

Run Gluten Clickhouse CI

@ulysses-you
Copy link
Copy Markdown
Contributor Author

cc @FelixYBW @zhouyuan @yma11 can you help review this pr ? thank you

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jul 7, 2023

Run Gluten Clickhouse CI

@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Jul 9, 2023

@ulysses-you thank you for improving on this! I tried this patch locally and it seems working, just two questions:

  • if the backend is not providing the events, will the UI show empty results? I'm asking b/c CK backend is not tested and it may not provide these events/metrics
  • This also work on AQE enabled case, right?

-yuan

@ulysses-you
Copy link
Copy Markdown
Contributor Author

thank you @zhouyuan for the thoughs,

if the backend is not providing the events, will the UI show empty results? I'm asking b/c CK backend is not tested and it may not provide these events/metrics

I think it works with ch backend. All backends use the same code path, since the main code chang is at GlutenDriverPlugin and columnar rule. It should have no difference between velox and ch backend.

This also work on AQE enabled case, right?

yes, it's right. This pr is compatible with AQE as it follows the behavior of vanilla Spark event SparkListenerSQLAdaptiveExecutionUpdate.

Copy link
Copy Markdown
Member

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
there are some TODOs but we could fix those in the next PRs.

@zhouyuan zhouyuan merged commit 71ad00c into apache:main Jul 10, 2023
@ulysses-you ulysses-you deleted the fallback-event branch July 11, 2023 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] Make fallback details visualize and analyzable

3 participants