Skip to content

[SPARK-55508][BUILD] Upgrade compress-lzf to 1.2.0 #54292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 3 commits into from
+16 −16
Closed

[SPARK-55508][BUILD] Upgrade compress-lzf to 1.2.0 #54292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c727084
[SPARK-55508][BUILD] Upgrade `compress-lzf` to 1.2.0
dongjoon-hyun Feb 13, 2026
329bb59
Benchmark results for org.apache.spark.io.LZFBenchmark (JDK 17, Scala…
dongjoon-hyun Feb 13, 2026
132adc3
Benchmark results for org.apache.spark.io.LZFBenchmark (JDK 21, Scala…
dongjoon-hyun Feb 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions core/benchmarks/LZFBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Benchmark LZFCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compress small objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Compression 256000000 int values in parallel 594 606 11 430.9 2.3 1.0X
Compression 256000000 int values single-threaded 570 572 1 449.0 2.2 1.0X
Compression 256000000 int values in parallel 604 611 9 423.7 2.4 1.0X
Compression 256000000 int values single-threaded 560 561 2 457.5 2.2 1.1X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compress large objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
Compression 1024 array values in 2 threads 23 25 1 0.0 22108.7 1.0X
Compression 1024 array values single-threaded 32 33 0 0.0 31296.0 0.7X
Compression 1024 array values in 1 threads 47 54 4 0.0 45412.7 1.0X
Copy link
Copy Markdown
Member

@pan3793 pan3793 Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this change expected?

Copy link
Copy Markdown
Member Author

@dongjoon-hyun dongjoon-hyun Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is consistent with Java 17 result. For Java 17, Compression 1024 array values in 1 threads is recorded before and after, @pan3793 .

Copy link
Copy Markdown
Member Author

@dongjoon-hyun dongjoon-hyun Feb 13, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, it looks like a result of non-deterministic benchmark code instead of bumping the version.

spark/core/src/test/scala/org/apache/spark/io/LZFBenchmark.scala

Lines 69 to 77 in 991fdc5

def getNThreads: Int = {
var nThreads = Runtime.getRuntime.availableProcessors
val jmx = ManagementFactory.getOperatingSystemMXBean
if (jmx != null) {
val loadAverage = jmx.getSystemLoadAverage.toInt
if (nThreads > 1 && loadAverage >= 1) nThreads = Math.max(1, nThreads - loadAverage)
}
nThreads
}

Copy link
Copy Markdown
Member

@pan3793 pan3793 Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, seems it was obsolete

Copy link
Copy Markdown
Member Author

@dongjoon-hyun dongjoon-hyun Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'ManagementFactory.getOperatingSystemMXBean().getSystemLoadAverage()' is the real time value in the GitHub Action, isn't it?

Copy link
Copy Markdown
Member

@pan3793 pan3793 Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a little bit hacky ...

getSystemLoadAverage returns the system load average for the last minute. In benchmark cases, the last minute load will be high because it was compiling the code

Copy link
Copy Markdown
Member Author

@dongjoon-hyun dongjoon-hyun Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, I agree with you. We may had better simply choose a constant value, 2, always.

Copy link
Copy Markdown
Member Author

@dongjoon-hyun dongjoon-hyun Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally, Kent gave us a result from M2 Max but GitHub Action resource is very smaller than that.

[info] OpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
[info] Apple M2 Max
[info] Compress large objects:                        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] -----------------------------------------------------------------------------------------------------------------------------
[info] Compression 1024 array values in 7 threads                12             13           1          0.1       11788.2       1.0X
[info] Compression 1024 array values single-threaded             23             23           0          0.0       22512.7       0.5X

Compression 1024 array values single-threaded 33 33 0 0.0 31870.0 1.4X


16 changes: 8 additions & 8 deletions core/benchmarks/LZFBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Benchmark LZFCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Compress small objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Compression 256000000 int values in parallel 593 602 8 431.4 2.3 1.0X
Compression 256000000 int values single-threaded 619 625 4 413.4 2.4 1.0X
Compression 256000000 int values in parallel 577 587 15 443.3 2.3 1.0X
Compression 256000000 int values single-threaded 572 573 1 447.9 2.2 1.0X

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Compress large objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
Compression 1024 array values in 1 threads 41 49 6 0.0 40204.2 1.0X
Compression 1024 array values single-threaded 33 33 1 0.0 32005.2 1.3X
Compression 1024 array values in 1 threads 46 49 1 0.0 44773.9 1.0X
Compression 1024 array values single-threaded 34 35 1 0.0 33331.6 1.3X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ commons-lang3/3.20.0//commons-lang3-3.20.0.jar
commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.15.0//commons-text-1.15.0.jar
compress-lzf/1.1.2//compress-lzf-1.1.2.jar
compress-lzf/1.2.0//compress-lzf-1.2.0.jar
curator-client/5.9.0//curator-client-5.9.0.jar
curator-framework/5.9.0//curator-framework-5.9.0.jar
curator-recipes/5.9.0//curator-recipes-5.9.0.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -888,7 +888,7 @@
<dependency>
<groupId>com.ning</groupId>
<artifactId>compress-lzf</artifactId>
<version>1.1.2</version>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
Expand Down