[VL] Unsupported spark function list [please leave a comment if you plan to pick some]

[philo-he]

Description

Here listed spark functions still not supported by Gluten Velox backend. Please leave a comment if you'd like to pick some. In the below list, [√] means someone is working in progress for the corresponding function.
You can find all functions' support status from this gluten doc.

To avoid duplicate work, before starting, please check whether a PR has been submitted in Velox community or whether it has already been implemented in Velox who holds most sql functions in its sparksql folder & prestosql folder.

Reference:

• spark sql expression
• spark built-in functions

---

• percentile_approx/approx_percentile (WIP, guangxin)
• concat_ws (PR ready, feat: Add Spark concat_ws function facebookincubator/velox#8854)
• unix_timestamp: "Only supports string type, with session timezone considered, todo: support date type"
• locate
• parse_url (PR drafted, not merged)
• urldecoder: "UDF, supported by spark as a built-in function since 3.4.0."
• normalizenanandzero
• arrayintersects
• default.json_split (udf, no need to impl.): "external UDF"
• parsejsonarray: "external UDF"
• struct
• percentile (@Yohahaha)
• first/first_value (@JkSelf)
• last/last_value (@JkSelf)
• posexplode (WIP, @marin-ma)
• trunc (WIP, HannanKan)
• months_between (PR ready)
• stack
• grouping_id
• printf (@Surbhi-Vijay)
• space (WIP, rhh777)
• inline (WIP, @marin-ma)
• to_unix_timestamp: "Only supports string type, with session timezone considered. todo: support date type"
• from_csv (@mudit-97)
• from_json (feat: Add Spark from_json function facebookincubator/velox#11709)
• to_json (@wecharyu)
• json_object_keys (@dcoliversun)
• json_tuple
• schema_of_csv
• schema_of_json
• to_csv
• make_ym_interval (WIP, @marin-ma)
• make_timestamp (WIP, @marin-ma)
• make_interval
• make_dt_interval
• from_utc_timestamp (@acvictor)
• extract
• exists (@lyy-pineapple)
• date_part
• zip_with
• transform (@Yohahaha)
• transform_keys
• transform_values
• map_from_entries (WIP, MaYan)
• map_filter (WIP, MaYan)
• map_entries (Done, by MaYan)
• forall (@lyy-pineapple)
• flatten (@ivoson)
• filter
• filter (array) (@ivoson)
• width_bucket
• array_sort (@boneanxs)
• xpath
• xpath_boolean
• xpath_double
• xpath_float
• xpath_int
• xpath_long
• xpath_number
• xpath_short
• xpath_string
• unbase64 (WIP, @fyp711)
• decode (partially supported if translated to caseWhen. WIP Cody)
• initcap (WIP, velox PR: 8676)
• unix_date (velox PR 8725, completed)
• count_min_sketch
• bool_and/every (@mskapilks)
• bool_or/any/some (@mskapilks)
• shuffle (completed)
• bround (@xumingming)
• format_string (@gaoyangxiaozhu)
• format_number (@gaoyangxiaozhu)
• soundex (@zhli1142015)
• levenshtein (@zhli1142015)
• cot (@honeyhexin)
• expm1 (@Donvi)
• stack (generator function, @xumingming)
• randn (@Donvi)
• empty2null (internal function, @jinchengchenghh)
• toprettystring (internal function, @jinchengchenghh)
• AtLeastNNonNulls (internal funciton, @zhli1142015)
• GetStructField (internal funciton)
• approx_count_distinct_for_intervals

• Since Spark-3.3 (related to ML, low priority)

• regr_count
• regr_avgx
• regr_avgy
• regr_r2
• regr_sxx
• regr_sxy
• regr_syy
• regr_slope
• regr_intercept

• Since Spark-3.3

• Since Spark-3.4

• mode
• get (@Yohahaha)
• array_append (@ivoson)
• array_insert (@ivoson)
• mode (@zhli1142015)

[Yohahaha]

I'd like support hex and unhex.

update: hex and unhex has already supported in Gluten.

[zwangsheng]

Hi i'd like to give a try with hour function.

[konjac]

Hi, I'd like to have a look into map_keys

[fyp711]

Hi I'd like to support find_in_set in velox

[HannanKan]

Hi, I'd like to support date_trunc/trunc.

[JkSelf]

Hi, I'd like to support dense_rank.

[JkSelf]

dense_rank already supported in velox facebookincubator/velox#6289.

[zhztheplayer]

• percentile_approx

• approx_percentile: "Third argument accuracy is different with velox, velox is double but spark is long"

The two stand for the same function I assume? I'll take these two if nobody is working on it.

[philo-he]

• percentile_approx

• approx_percentile: "Third argument accuracy is different with velox, velox is double but spark is long"

The two stand for the same function I assume? I'll take these two if nobody is working on it.

Yes, they are one thing. Just unify them into one checkbox. Thanks!

[JkSelf]

I will take a look ntile window function.

[zhouyuan]

ubase64:
#4482