Skip to content

[SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark#54540

Closed
pan3793 wants to merge 3 commits intoapache:masterfrom
pan3793:SPARK-55744
Closed

[SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark#54540
pan3793 wants to merge 3 commits intoapache:masterfrom
pan3793:SPARK-55744

Conversation

@pan3793
Copy link
Copy Markdown
Member

@pan3793 pan3793 commented Feb 27, 2026

What changes were proposed in this pull request?

Fix V2FunctionBenchmark, caused by SPARK-53968 (#52681)

$ build/sbt "sql/Test/runMain org.apache.spark.sql.connector.functions.V2FunctionBenchmark"
...
[info] running (fork) org.apache.spark.sql.connector.functions.V2FunctionBenchmark
[error] WARNING: Using incubator modules: jdk.incubator.vector
[info] 13:43:41.084 WARN org.apache.spark.util.Utils: Your hostname, H27212-MAC-01.local, resolves to a loopback address: 127.0.0.1; using 10.242.159.140 instead (on interface en0)
[info] 13:43:41.087 WARN org.apache.spark.util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
[info] 13:43:41.260 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] Running benchmark: scalar function (long + long) -> long, result_nullable = true codegen = true
[info]   Running case: native_long_add
[error] Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.catalyst.expressions.NumericEvalContext.evalMode()" because the return value of "org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.evalContext()" is null
[error] 	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.evalMode(arithmetic.scala:201)
[error] 	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.failOnError(arithmetic.scala:209)
[error] 	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.initQueryContext(arithmetic.scala:251)
[error] 	at org.apache.spark.sql.catalyst.expressions.SupportQueryContext.$init$(Expression.scala:670)
[error] 	at org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.<init>(arithmetic.scala:193)
[error] 	at org.apache.spark.sql.connector.functions.V2FunctionBenchmark$NativeAdd.<init>(V2FunctionBenchmark.scala:111)
[error] 	at org.apache.spark.sql.connector.functions.V2FunctionBenchmark$.$anonfun$scalarFunctionBenchmark$3(V2FunctionBenchmark.scala:88)
...

Why are the changes needed?

Fix the benchmark.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Verified locally, GHA benchmark reports for JDK 17 and 21 are updated.

Was this patch authored or co-authored using generative AI tooling?

No.

Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, there is a JIRA issue, SPARK-55519 already for this, @pan3793 . Could you use that by change the PR title and close the JIRA issue, SPARK-55744, because we prefer to use the smaller JIRA IDs for duplications?

It's reported here.

@pan3793
Copy link
Copy Markdown
Member Author

pan3793 commented Feb 27, 2026

@dongjoon-hyun thanks for the information, was not aware of that, let me use that and close SPARK-55744 as duplicated

@pan3793 pan3793 changed the title [SPARK-55744][SQL][TESTS] Fix V2FunctionBenchmark [SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark Feb 27, 2026
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @pan3793 .

…tionBenchmark (JDK 21, Scala 2.13, split 1 of 1)
…tionBenchmark (JDK 17, Scala 2.13, split 1 of 1)
@pan3793
Copy link
Copy Markdown
Member Author

pan3793 commented Feb 27, 2026

Benchmark results for JDK 17 and 21 are updated, and they look reasonable. @dongjoon-hyun, could you please double-check too?

JDK 25 benchmark results will be uploaded in bulk later, via an independent PR.

@dongjoon-hyun
Copy link
Copy Markdown
Member

All results look correct to me. Thank you, @pan3793 . Merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants