Skip to content

[EPIC] Add Spark expression coverage #240

@viirya

Description

@viirya

What is the problem the feature request solves?

This is an umbrella ticket for the list of unsupported Spark expressions. This is not necessary comprehensive list of all Spark expressions because they are too many. We can start from frequently used expressions.

Hash expressions #205

  • Md5: supported by DataFusion, but Comet disables it for now. DataFusion crypto_expressions includes blake3 which cannot be built on Mac platform. We cannot separately enable only md5 in DataFusion.
  • Murmur3Hash
  • XxHash64
  • HiveHash
  • Sha1
  • Sha2
  • Crc32

Datetime expressions

Conditional expressions

  • If
  • CaseWhen

Arithmetic expressions

  • UnaryMinus
  • UnaryPositive
  • Abs
  • Add
  • Subtract
  • Multiply
  • Divide
  • IntegralDivide
  • Remainder
  • Pmod
  • Least
  • Greatest

Bitwise expressions

  • BitwiseAnd
  • BitwiseOr
  • BitwiseXor
  • BitwiseNot
  • BitwiseCount
  • BitwiseGet

String expressions

  • Ascii
  • BitLength
  • StringInstr
  • StringRepeat
  • StringReplace
  • StringTranslate
  • StringTrim
  • StringTrimLeft
  • StringTrimRight
  • StringTrimBoth
  • Upper
  • Lower
  • Length
  • InitCap
  • Chr
  • ConcatWs

Math expressions

  • Acos
  • Asin
  • Atan
  • Atan2
  • Ceil
  • Cos
  • Exp
  • Floor
  • Log
  • Log10
  • Log2
  • Pow
  • Round (3.3+)
  • Signum
  • Sin
  • Sqrt
  • Tan
  • Asinh
  • Sinh
  • Csc
  • Rint
  • Log1p
  • Factorial
  • RoundFloor
  • Expm1
  • Conv
  • Acosh
  • Cosh
  • Sec
  • RoundCeil
  • Cbrt
  • Pi
  • EulerNumber
  • Cot
  • Tanh
  • Atanh
  • ToDegrees
  • ToRadians
  • Bin
  • Hex
  • Unhex
  • ShiftLeft
  • ShiftRight
  • ShiftRightUnsigned
  • Hypot
  • Logarithm
  • BRound
  • WidthBucket

Predicates

  • In
  • InSet
  • Not
  • InSubquery
  • And
  • Or
  • EqualTo
  • EqualNullSafe
  • EqualNull
  • LessThan
  • LessThanOrEqual
  • GreaterThan
  • GreaterThanOrEqual

Null expressions

  • Coalesce

Aggregate expressions

Others

  • Cast

...

Describe the potential solution

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions