Skip to content

[CORE] Bump iceberg version of spark 3.3 to 1.5.0#8418

Merged
zhztheplayer merged 7 commits intoapache:mainfrom
j7nhai:j7nhai-iceberg-dev
Jan 23, 2025
Merged

[CORE] Bump iceberg version of spark 3.3 to 1.5.0#8418
zhztheplayer merged 7 commits intoapache:mainfrom
j7nhai:j7nhai-iceberg-dev

Conversation

@j7nhai
Copy link
Copy Markdown
Contributor

@j7nhai j7nhai commented Jan 3, 2025

What changes were proposed in this pull request?

Bump iceberg version of spark 3.3 to 1.5.0

How was this patch tested?

UT

@github-actions github-actions bot added CORE works for Gluten Core VELOX DATA_LAKE labels Jan 3, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 3, 2025

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 3, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 3, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 6, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 6, 2025

Run Gluten Clickhouse CI on x86

@j7nhai
Copy link
Copy Markdown
Contributor Author

j7nhai commented Jan 6, 2025

The PR enables spark-3.3 to upgrade iceberg's version.

It seems this change is unrelated to ClickHouse, but why ClickHouse CI fail ? @philo-he

@philo-he
Copy link
Copy Markdown
Member

philo-he commented Jan 7, 2025

Run Gluten Clickhouse CI on x86

@philo-he
Copy link
Copy Markdown
Member

philo-he commented Jan 7, 2025

The PR enables spark-3.3 to upgrade iceberg's version.

It seems this change is unrelated to ClickHouse, but why ClickHouse CI fail ? @philo-he

Not sure. Just retriggered the CI.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 7, 2025

Run Gluten Clickhouse CI on x86

@philo-he
Copy link
Copy Markdown
Member

philo-he commented Jan 7, 2025

@j7nhai, could you check the error log? See public account for login: https://github.com/apache/incubator-gluten/blob/main/docs/get-started/ClickHouse.md#new-ci-system.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 7, 2025

Run Gluten Clickhouse CI on x86

@j7nhai
Copy link
Copy Markdown
Contributor Author

j7nhai commented Jan 7, 2025

@j7nhai, could you check the error log? See public account for login: https://github.com/apache/incubator-gluten/blob/main/docs/get-started/ClickHouse.md#new-ci-system.

Thanks, I have fixed it.

@zhouyuan zhouyuan changed the title Bump iceberg version of spark 3.3 to 1.5.0 [CORE] Bump iceberg version of spark 3.3 to 1.5.0 Jan 8, 2025
Comment thread pom.xml
<spark.version>3.3.1</spark.version>
<!-- keep using iceberg v1.3.1 for parquet compatibilty. -->
<iceberg.version>1.3.1</iceberg.version>
<iceberg.version>1.5.0</iceberg.version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this parquet compatibility is still valid for spark33
CC @ulysses-you

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what the compatibility issue is here. @yma11

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I remember, there should be an API mismatch because of different parquet version used by icerberg 1.5.0+ and Spark3.3.1. But I am not sure why it's okay now as no UT failure detected. Suggest you guys double check this part.

Copy link
Copy Markdown
Member

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@zhztheplayer zhztheplayer merged commit f0336c0 into apache:main Jan 23, 2025
@GlutenPerfBot
Copy link
Copy Markdown
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_01_23_2025_time.csv log/native_master_01_22_2025_e329341e39_time.csv difference percentage
q1 16.30 15.99 -0.315 98.07%
q2 15.72 16.26 0.541 103.44%
q3 3.07 4.87 1.805 158.81%
q4 86.81 85.95 -0.866 99.00%
q5 12.05 13.31 1.263 110.48%
q6 5.94 3.89 -2.053 65.46%
q7 8.06 6.71 -1.354 83.21%
q8 4.81 5.06 0.243 105.05%
q9 28.05 28.29 0.246 100.88%
q10 12.73 13.14 0.402 103.16%
q11 42.90 43.20 0.305 100.71%
q12 2.15 2.50 0.356 116.56%
q13 8.84 10.47 1.636 118.52%
q14a 67.94 69.52 1.579 102.32%
q14b 58.85 60.91 2.065 103.51%
q15 3.75 3.62 -0.121 96.76%
q16 30.07 29.53 -0.539 98.21%
q17 7.43 7.70 0.269 103.63%
q18 10.58 9.75 -0.827 92.18%
q19 3.46 5.44 1.973 156.95%
q20 2.26 2.29 0.021 100.93%
q21 1.89 1.86 -0.023 98.79%
q22 10.16 9.51 -0.646 93.64%
q23a 141.16 139.49 -1.664 98.82%
q23b 162.48 161.50 -0.981 99.40%
q24a 105.19 98.53 -6.658 93.67%
q24b 99.34 92.46 -6.881 93.07%
q25 7.46 6.58 -0.883 88.17%
q26 5.39 3.89 -1.501 72.16%
q27 5.15 5.14 -0.009 99.83%
q28 39.96 36.05 -3.919 90.19%
q29 19.52 19.85 0.324 101.66%
q30 7.01 6.34 -0.679 90.32%
q31 10.19 10.71 0.516 105.06%
q32 2.82 2.22 -0.604 78.60%
q33 7.57 7.45 -0.126 98.33%
q34 4.26 6.41 2.149 150.46%
q35 10.98 11.51 0.525 104.78%
q36 5.66 5.67 0.005 100.08%
q37 5.46 6.04 0.576 110.54%
q38 17.53 24.50 6.968 139.75%
q39a 4.56 6.18 1.611 135.31%
q39b 6.23 4.70 -1.524 75.53%
q40 6.46 5.41 -1.052 83.71%
q41 0.88 0.85 -0.031 96.51%
q42 1.21 1.22 0.015 101.21%
q43 4.76 4.46 -0.293 93.83%
q44 11.69 11.97 0.282 102.41%
q45 4.19 4.34 0.149 103.55%
q46 5.56 5.28 -0.279 94.98%
q47 20.59 20.98 0.397 101.93%
q48 6.39 6.37 -0.020 99.68%
q49 10.53 9.97 -0.567 94.62%
q50 39.04 39.39 0.350 100.90%
q51 14.56 13.97 -0.583 96.00%
q52 1.19 1.28 0.090 107.55%
q53 3.71 3.41 -0.295 92.03%
q54 6.84 7.03 0.182 102.66%
q55 1.57 1.24 -0.331 78.94%
q56 7.43 7.61 0.183 102.46%
q57 12.27 12.67 0.396 103.23%
q58 3.35 3.92 0.573 117.12%
q59 6.65 7.18 0.524 107.88%
q60 7.99 8.64 0.647 108.10%
q61 7.55 8.03 0.480 106.36%
q62 5.16 5.05 -0.102 98.02%
q63 3.14 3.47 0.324 110.30%
q64 62.09 59.26 -2.833 95.44%
q65 29.93 29.71 -0.228 99.24%
q66 4.62 4.33 -0.292 93.69%
q67 228.90 225.43 -3.474 98.48%
q68 4.15 4.75 0.594 114.31%
q69 9.20 7.01 -2.183 76.26%
q70 12.68 13.06 0.382 103.01%
q71 4.43 4.67 0.243 105.49%
q72 38.19 40.93 2.736 107.17%
q73 3.60 3.26 -0.348 90.34%
q74 27.01 27.99 0.978 103.62%
q75 43.89 43.73 -0.156 99.64%
q76 15.32 14.18 -1.143 92.54%
q77 3.49 3.44 -0.053 98.49%
q78 84.96 84.36 -0.604 99.29%
q79 5.18 4.86 -0.326 93.70%
q80 16.60 17.87 1.268 107.64%
q81 8.67 8.47 -0.200 97.69%
q82 10.54 10.29 -0.257 97.56%
q83 2.66 2.79 0.138 105.20%
q84 3.83 3.73 -0.107 97.20%
q85 9.79 9.85 0.060 100.61%
q86 4.53 4.59 0.064 101.42%
q87 17.90 18.01 0.114 100.64%
q88 24.14 22.66 -1.486 93.85%
q89 4.55 4.17 -0.378 91.70%
q90 3.60 3.55 -0.059 98.37%
q91 4.98 5.12 0.139 102.78%
q92 2.23 2.73 0.498 122.34%
q93 55.08 54.29 -0.790 98.57%
q94 16.06 18.99 2.927 118.22%
q9 95.97 98.32 2.348 102.45%
q5 3.16 3.20 0.041 101.29%
q96 27.88 29.36 1.480 105.31%
q97 2.84 2.77 -0.065 97.71%
q98 10.21 9.89 -0.314 96.93%
q99 10.21 9.89 -0.314 96.93%
total 2207.35 2200.31 -7.043 99.68%

@GlutenPerfBot
Copy link
Copy Markdown
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_01_23_2025_time.csv log/native_master_01_22_2025_e329341e39_time.csv difference percentage
q1 44.10 43.03 -1.068 97.58%
q2 43.28 43.93 0.645 101.49%
q3 92.69 91.97 -0.721 99.22%
q4 67.94 70.49 2.551 103.75%
q5 183.59 181.93 -1.661 99.10%
q6 19.27 18.14 -1.128 94.15%
q7 107.17 104.54 -2.626 97.55%
q8 186.90 187.11 0.209 100.11%
q9 282.55 284.03 1.477 100.52%
q10 101.04 105.53 4.497 104.45%
q11 34.17 34.73 0.558 101.63%
q12 42.07 44.13 2.054 104.88%
q13 75.61 76.60 0.990 101.31%
q14 35.83 35.69 -0.139 99.61%
q15 66.80 67.16 0.364 100.54%
q16 26.84 28.04 1.200 104.47%
q17 233.26 233.96 0.697 100.30%
q18 358.63 349.07 -9.556 97.34%
q19 38.36 36.68 -1.677 95.63%
q20 59.86 59.49 -0.374 99.38%
q21 534.83 528.54 -6.283 98.83%
q22 24.68 24.54 -0.143 99.42%
total 2659.47 2649.33 -10.134 99.62%

baibaichen pushed a commit to baibaichen/gluten that referenced this pull request Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants