feat(spark): add trunc, date_trunc and time_trunc functions by cht42 · Pull Request #19829 · apache/datafusion

cht42 · 2026-01-15T12:00:30Z

Which issue does this PR close?

Rationale for this change

implement spark:

What changes are included in this PR?

Add spark compatible wrappers around datafusion date_trunc function to handle spark specificities.

Are these changes tested?

Yes in SLT

Are there any user-facing changes?

Yes

Jefffrey · 2026-01-15T12:39:41Z

+
+        Ok(ExprSimplifyResult::Simplified(Expr::ScalarFunction(
+            ScalarFunction::new_udf(
+                datafusion_functions::datetime::date_trunc(),


Just concerned about if matching return field nullability here is something we should watch for?

yep DF's date_trunc returm field will be nullable.. if #19511 goes through it should fix this issue

I don't see #19511 landing anytime soon so we might need to fix this in the DF date_trunc to ensure consistency here

martin-g · 2026-01-15T13:22:32Z

+        args: Vec<Expr>,
+        _info: &SimplifyContext,
+    ) -> Result<ExprSimplifyResult> {
+        let fmt_expr = &args[0];


Suggested change

let fmt_expr = &args[0];

let [fmt_expr, time_expr] = take_function_args(self.name(), args)?;

martin-g · 2026-01-15T13:24:24Z

+        }
+
+        Ok(ExprSimplifyResult::Simplified(Expr::ScalarFunction(
+            ScalarFunction::new_udf(datafusion_functions::datetime::date_trunc(), args),


fmt is normalized (lowercased) above and validated but here you pass the original args (non-normalized).
Maybe it will be better to pass the fmt:

let fmt_expr = Expr::Literal(ScalarValue::new_utf8(fmt.as_str()), None); ... ScalarFunction::new_udf( datafusion_functions::datetime::date_trunc(), vec![fmt_expr, time_expr], ),

shouldn't matter, DF will handle the original argument as well and lowercase it

andygrove · 2026-01-15T17:43:15Z

@cht42 could you add tests for DST handling with different time zones? I am not convinced that the current PR handles this correctly.

Here is an AI-generated test that highlights the issue:

# Test with explicit timezone in timestamp
statement ok
SET datafusion.execution.time_zone = 'America/New_York';

# Cross-day boundary: 03:30 UTC on July 15 = 23:30 EDT on July 14
# TODO: Spark returns 2024-07-14 (converts to session tz first)
# DataFusion returns 2024-07-15 (truncates in UTC)
query P
SELECT date_trunc('DAY', '2024-07-15T03:30:00Z'::timestamp);
----
2024-07-15T00:00:00

# Reset timezone for other tests
statement ok
RESET datafusion.execution.time_zone

Spark repro:

$ /opt/spark-3.5.7-bin-hadoop3/bin/spark-sql --conf spark.sql.session.timeZone=America/New_York -e "SELECT date_trunc('DAY', timestamp'2024-07-15T03:30:00Z');"
...
Spark master: local[*], Application Id: local-1768499563499
2024-07-14 00:00:00

…tion

cht42 · 2026-01-15T19:12:52Z

good catch @andygrove , I added the tests and some logic to handle those cases. let me know what you think

…handling

Jefffrey · 2026-01-16T13:45:10Z

        &self.signature
    }

+    // keep return_type implementation for information schema generation


This seems like a bug

datafusion/datafusion/catalog/src/information_schema.rs

Line 424 in 3ea21aa

// only handle the function which implemented [`ScalarUDFImpl::return_type`] method

Raised

Information schema should use return_field_from_args #19870

Jefffrey · 2026-01-16T13:46:50Z

+            // Timestamp without timezone: use as-is
+            _ => ts_expr,


The above arm catches all timestamps though? Since it doesn't match on Some(tz); so when would this arm take effect?

yes, updated to throw an error if we go in this arm, should not be happening

andygrove · 2026-01-16T16:23:55Z

good catch @andygrove , I added the tests and some logic to handle those cases. let me know what you think

Looks good. Thanks for addressing that.

…ent is not a Timestamp

Jefffrey · 2026-01-18T16:31:27Z

Looks like some conflicts to fix

cht42 · 2026-01-18T17:50:08Z

 }

 pub fn functions() -> Vec<Arc<ScalarUDF>> {
    vec![


changes are because i sorted alphabetically

cht42 · 2026-01-18T17:50:57Z

Looks like some conflicts to fix

fixed

Jefffrey · 2026-01-19T04:14:47Z

Thanks @cht42, @andygrove & @martin-g

…9829) ## Which issue does this PR close?  - Closes apache#19828. - Part of apache#15914 ## Rationale for this change implement spark: - https://spark.apache.org/docs/latest/api/sql/index.html#trunc - https://spark.apache.org/docs/latest/api/sql/index.html#date_trunc - https://spark.apache.org/docs/latest/api/sql/index.html#time_trunc ## What changes are included in this PR? Add spark compatible wrappers around datafusion date_trunc function to handle spark specificities. ## Are these changes tested? Yes in SLT ## Are there any user-facing changes? Yes

feat(spark): add trunc, date_trunc and time_trunc functions

a81ab4f

github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Jan 15, 2026

fix: typo in test

e214d65

Jefffrey reviewed Jan 15, 2026

View reviewed changes

martin-g reviewed Jan 15, 2026

View reviewed changes

andygrove reviewed Jan 15, 2026

View reviewed changes

Comment thread datafusion/spark/src/function/datetime/date_trunc.rs

cht42 added 3 commits January 15, 2026 21:56

fix: replace internal_err with plan_err in date and time trunc functions

9d7addf

fix

e402557

feat(spark): enhance date_trunc to handle session timezone for trunca…

120b9e9

…tion

refactor: add return_field_from_args for DateTruncFunc

3b6579e

github-actions Bot added the functions Changes to functions implementation label Jan 16, 2026

cht42 added 2 commits January 16, 2026 08:49

fix: fix information schema generation

48488eb

fix: update date_trunc to return UTC timestamps and improve timezone …

fb1f92a

…handling

Jefffrey reviewed Jan 16, 2026

View reviewed changes

fix: improve error handling for DATE_TRUNC function when second argum…

a6cbe76

…ent is not a Timestamp

Jefffrey approved these changes Jan 17, 2026

View reviewed changes

Merge branch 'main' into spark-trunc

54e2214

cht42 commented Jan 18, 2026

View reviewed changes

Jefffrey added this pull request to the merge queue Jan 19, 2026

Merged via the queue into apache:main with commit 4ed808a Jan 19, 2026
28 checks passed

	let fmt_expr = &args[0];
	let [fmt_expr, time_expr] = take_function_args(self.name(), args)?;

Conversation

cht42 commented Jan 15, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andygrove commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cht42 commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Jan 16, 2026

Uh oh!

Jefffrey commented Jan 18, 2026

Uh oh!

cht42 Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cht42 commented Jan 18, 2026

Uh oh!

Uh oh!

Jefffrey commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented Jan 15, 2026 •

edited

Loading

cht42 Jan 18, 2026 •

edited

Loading