29 Spark expressions are already implemented in the datafusion-spark crate but not yet wired up in Comet. Adding support for these is lower effort than typical expression work since the native implementation already exists — the main task is adding the JVM serde layer.
| Category |
Expressions |
| aggregate |
collect_list, try_sum |
| array |
shuffle, slice |
| bitwise |
shiftrightunsigned |
| datetime |
add_months, from_utc_timestamp, make_dt_interval, make_interval, to_utc_timestamp, unix_micros, unix_millis, unix_seconds |
| json |
json_tuple |
| map |
map_from_arrays, map_from_entries |
| math |
csc, factorial, pmod, rint, sec |
| string |
base64, elt, format_string, soundex, unbase64 |
| url |
parse_url, url_decode, url_encode |
29 Spark expressions are already implemented in the
datafusion-sparkcrate but not yet wired up in Comet. Adding support for these is lower effort than typical expression work since the native implementation already exists — the main task is adding the JVM serde layer.collect_list,try_sumshuffle,sliceshiftrightunsignedadd_months,from_utc_timestamp,make_dt_interval,make_interval,to_utc_timestamp,unix_micros,unix_millis,unix_secondsjson_tuplemap_from_arrays,map_from_entriescsc,factorial,pmod,rint,secbase64,elt,format_string,soundex,unbase64parse_url,url_decode,url_encode