Skip to content

[SPARK-37907][SQL] InvokeLike support ConstantFolding #35207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 9 commits into from
Closed

[SPARK-37907][SQL] InvokeLike support ConstantFolding #35207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c5f9a00
[SPARK-37907][SQL] StaticInvoke support ConstantFolding
AngersZhuuuu Jan 14, 2022
5815116
Update objects.scala
AngersZhuuuu Jan 14, 2022
36ccc12
Update CharVarcharTestSuite.scala
AngersZhuuuu Jan 14, 2022
aebdef8
Update objects.scala
AngersZhuuuu Jan 14, 2022
0fa6885
Update ConstantFoldingSuite.scala
AngersZhuuuu Jan 15, 2022
c9ff7f8
re--trigger
AngersZhuuuu Jan 20, 2022
8ed3229
Merge branch 'master' into SPARK-37907
AngersZhuuuu Jan 20, 2022
efa0070
Update objects.scala
AngersZhuuuu Jan 20, 2022
5cbd015
Update objects.scala
AngersZhuuuu Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ trait InvokeLike extends Expression with NonSQLExpression with ImplicitCastInput

def propagateNull: Boolean

override def foldable: Boolean = children.forall(_.foldable) && deterministic
Copy link
Contributor

@cloud-fan cloud-fan Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we remove deterministic? I don't see other expressions require it when defining foldable.

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we remove deterministic? I don't see other expressions require it when defining foldable.

If it's not deterministic, such as Invoke return a rand result, if we fold this, we may return incorrect answer

protected lazy val needNullCheck: Boolean = needNullCheckForIndex.contains(true)
protected lazy val needNullCheckForIndex: Array[Boolean] =
arguments.map(a => a.nullable && (propagateNull ||
Expand Down
40 changes: 40 additions & 0 deletions ...atalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,13 @@ import org.apache.spark.sql.catalyst.analysis.{EliminateSubqueryAliases, Unresol
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, NewInstance, StaticInvoke}
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.catalyst.util.GenericArrayData
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.ByteArray

class ConstantFoldingSuite extends PlanTest {

Expand Down Expand Up @@ -299,4 +302,41 @@ class ConstantFoldingSuite extends PlanTest {

comparePlans(optimized, correctAnswer)
}

test("SPARK-37907: InvokeLike support ConstantFolding") {
val originalQuery =
testRelation
.select(
StaticInvoke(
classOf[ByteArray],
BinaryType,
"lpad",
Seq(Literal("Spark".getBytes), Literal(7), Literal("W".getBytes)),
Seq(BinaryType, IntegerType, BinaryType),
returnNullable = false).as("c1"),
Invoke(
Literal.create("a", StringType),
"substring",
StringType,
Seq(Literal(0), Literal(1))).as("c2"),
NewInstance(
cls = classOf[GenericArrayData],
arguments = Literal.fromObject(List(1, 2, 3)) :: Nil,
inputTypes = Nil,
propagateNull = false,
dataType = ArrayType(IntegerType),
outerPointer = None).as("c3"))

val optimized = Optimize.execute(originalQuery.analyze)

val correctAnswer =
testRelation
.select(
Literal("WWSpark".getBytes()).as("c1"),
Literal.create("a", StringType).as("c2"),
Literal.create(new GenericArrayData(List(1, 2, 3)), ArrayType(IntegerType)).as("c3"))
.analyze

comparePlans(optimized, correctAnswer)
}
}
4 changes: 2 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,8 +332,8 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
sql(s"CREATE TABLE t(c STRUCT<c: $typeName(5)>) USING $format")
sql("INSERT INTO t SELECT struct(null)")
checkAnswer(spark.table("t"), Row(Row(null)))
val e = intercept[SparkException](sql("INSERT INTO t SELECT struct('123456')"))
assert(e.getCause.getMessage.contains(s"Exceeds char/varchar type length limitation: 5"))
val e = intercept[RuntimeException](sql("INSERT INTO t SELECT struct('123456')"))
assert(e.getMessage.contains(s"Exceeds char/varchar type length limitation: 5"))
}
}

Expand Down