feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation #19142

ethan-tyler · 2025-12-07T08:33:25Z

Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations. Changes:

Add TableProvider::delete_from() method for DELETE operations
Add TableProvider::update() method for UPDATE operations
Wire physical planner to route DML operations to TableProvider
Implement DELETE/UPDATE for MemTable as reference implementation
Add comprehensive sqllogictest coverage

This provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners.

Which issue does this PR close?

Closes Support delete_from and update in TableProvider #16959
Related to How to support deletes? #12406

Rationale for this change

Datafusion parses DELETE/UPDATE but returns NotImplemented("Unsupported logical plan: Dml(Delete)") at physical planning. Downstream projects (iceberg-rust, delta-rs) must implement custom planners to work around this.

What changes are included in this PR?

Adds TableProvider hooks for row-level DML:

delete_from(state, filters) - deletes rows matching filter predicates
update(state, assignments, filters) - updates matching rows with new values

Physical planner routes WriteOp::Delete and WriteOp::Update to these methods. Tables that don't support DML return NotImplemented (the default behavior). MemTable reference implementation demonstrates:

Filter evaluation with SQL three-valued logic (NULL predicates don't match)
Multi-column updates with expression evaluation
Proper row count reporting

Are these changes tested?

Yes:

Unit tests for physical planning: cargo test -p datafusion --test custom_sources_cases
SQL logic tests: dml_delete.slt and dml_update.slt with comprehensive coverage

Are there any user-facing changes?

New trait methods on TableProvider:

async fn delete_from(&self, state: &dyn Session, filters: Vec<Expr>) -> Result<Arc<dyn ExecutionPlan>>;
async fn update(&self, state: &dyn Session, assignments: Vec<(String, Expr)>, filters: Vec<Expr>) -> Result<Arc<dyn ExecutionPlan>>;

Fully backward compatible. Default implementations return NotImplemented.

ethan-tyler · 2025-12-07T16:52:05Z

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.

cc @alamb @adriangb

Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations. Changes: - Add `DmlCapabilities` struct to declare DELETE/UPDATE support - Add `TableProvider::dml_capabilities()` method (defaults to NONE) - Add `TableProvider::delete_from()` method for DELETE operations - Add `TableProvider::update()` method for UPDATE operations - Wire physical planner to route DML operations to TableProvider - Add helper functions for extracting filters and assignments This provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners. A reference MemTable implementation follows in a subsequent PR.

Clippy warns about assertions on constants being optimized away. Changed to assert_eq! for explicit value comparison.

Add complete DELETE and UPDATE support to MemTable as a reference implementation for the TableProvider DML hooks. Implementation: - `delete_from()`: Evaluate filter predicates, remove matching rows - `update()`: Evaluate filters, apply SET expressions to matching rows - `DmlResultExec`: ExecutionPlan returning affected row count - Correct SQL NULL semantics (NULL predicates preserve rows) Tests (812 lines): - DELETE: basic, all rows, predicates (=, >, AND, OR, LIKE, BETWEEN, IN, NOT IN, IS NULL, IS NOT NULL), NULL handling, functions, types - UPDATE: basic, SET expressions, multi-column, CASE, functions, column references, NULL handling, error cases All sqllogictest tests pass: cargo test --test sqllogictests -- dml_delete dml_update

Now that MemTable supports DELETE and UPDATE, the existing EXPLAIN tests produce physical plans instead of "not implemented" errors. Updated expected output to reflect the new DmlResultExec execution.

ethan-tyler · 2025-12-07T18:41:18Z

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.

cc @alamb @adriangb

Rebased on latest main. Fixed clippy warnings in tests and updated delete.slt/update.slt expected output to reflect new DmlResultExec physical plans. All tests pass locally. Could you please re-trigger the CI? Thanks!

Feat/tableprovider dml hooks

Feat/memtable dml impl

Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations. Changes: - Add `DmlCapabilities` struct to declare DELETE/UPDATE support - Add `TableProvider::dml_capabilities()` method (defaults to NONE) - Add `TableProvider::delete_from()` method for DELETE operations - Add `TableProvider::update()` method for UPDATE operations - Wire physical planner to route DML operations to TableProvider - Add helper functions for extracting filters and assignments This provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners. A reference MemTable implementation follows in a subsequent PR.

Clippy warns about assertions on constants being optimized away. Changed to assert_eq! for explicit value comparison.

- Change assert_eq!(x, true/false) to assert!(x)/assert!(!x) per clippy - Update delete.slt and update.slt expected error messages to match new physical planner behavior for unsupported DML operations

Add complete DELETE and UPDATE support to MemTable as a reference implementation for the TableProvider DML hooks. Implementation: - `delete_from()`: Evaluate filter predicates, remove matching rows - `update()`: Evaluate filters, apply SET expressions to matching rows - `DmlResultExec`: ExecutionPlan returning affected row count - Correct SQL NULL semantics (NULL predicates preserve rows) Tests (812 lines): - DELETE: basic, all rows, predicates (=, >, AND, OR, LIKE, BETWEEN, IN, NOT IN, IS NULL, IS NOT NULL), NULL handling, functions, types - UPDATE: basic, SET expressions, multi-column, CASE, functions, column references, NULL handling, error cases All sqllogictest tests pass: cargo test --test sqllogictests -- dml_delete dml_update

Now that MemTable supports DELETE and UPDATE, the existing EXPLAIN tests produce physical plans instead of "not implemented" errors. Updated expected output to reflect the new DmlResultExec execution.

ethan-tyler · 2025-12-08T00:52:27Z

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.
cc @alamb @adriangb

Rebased on latest main. Fixed clippy warnings in tests and updated delete.slt/update.slt expected output to reflect new DmlResultExec physical plans. All tests pass locally. Could you please re-trigger the CI? Thanks!

Rebased on main. Previous CI failures were due to clippy bool_assert_comparison warnings and mismatched expected error messages in test files. Requesting a CI trigger to confirm I've successfully stopped embarrassing myself. 🙏

alamb

Thank you for this PR @ethan-tyler 🙏

I have a few suggestions for API and then I think this PR needs some tests for the new planning code

Otherwise this is looking very nice 👌

alamb · 2025-12-08T20:21:47Z

datafusion/core/src/physical_planner.rs

+            }) => {
+                if let Some(provider) =
+                    target.as_any().downcast_ref::<DefaultTableSource>()
+                {
+                    let capabilities = provider.table_provider.dml_capabilities();
+                    if !capabilities.delete {
+                        return plan_err!(
+                            "Table '{}' does not support DELETE operations",
+                            table_name
+                        );
+                    }
+                    let filters = extract_dml_filters(input)?;
+                    provider
+                        .table_provider
+                        .delete_from(session_state, filters)
+                        .await?
+                } else {
+                    return exec_err!(
+                        "Table source can't be downcasted to DefaultTableSource"
+                    );
+                }
+            }


I think a better (smaller API surface) would be to simply try and call provider.delete_from(session_state, filters) and pass along the context

Then you could remove DmlCapabilities entirely

Something like:

LogicalPlan::Dml(DmlStatement { table_name, target, op: WriteOp::Delete, input, .. }) => { if let Some(provider) = target.as_any().downcast_ref::<DefaultTableSource>() { let filters = extract_dml_filters(input)?; provider .table_provider .delete_from(session_state, filters) .await .context(format!( "DELETE operation on table '{}'", table_name ))? } }

alamb · 2025-12-08T20:24:11Z

datafusion/expr/src/logical_plan/dml.rs


+/// DML operations supported by a table.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)]
+pub struct DmlCapabilities {


I am not sure this API is necessary (see above for alternate)

alamb · 2025-12-08T20:29:28Z

datafusion/core/src/physical_planner.rs

    Ok((physical_expr, physical_name))
 }

+/// Extract filter predicates from a DML input plan (DELETE/UPDATE).


I think we need some tests for the planning of delete and update -- specifically that the columns extracted are correct

Perhaps somewhere in https://github.com/apache/datafusion/blob/2db3aeaa53c2fe7cbc709ad517fd6255d5f02074/datafusion/core/tests/custom_sources_cases/mod.rs#L215-L214

The custom provider doesn't actually need to delete anything, just print out the columns that it was passed, for example

ethan-tyler · 2025-12-09T01:17:36Z

Thanks so much for the review @alamb! Really appreciate you taking the time And yeah, you're totally right about the redudent DmlCapabilities. The default impl already returns the error, so the capability check is basically just... checking if we're going to get an error before we get the error. lol.

Will rip that out and simplify the planner logic. Also adding .context() for better errors and I'll throw in some tests for the filter/assignment extraction, was thinking a mock TableProvider that just captures what gets passed to it so we can assert on it.

Should have something pushed soon. Thanks again for taking the time on this!

- Remove DmlCapabilities struct and dml_capabilities() trait method - Simplify physical planner to try-call pattern with .context() errors - Add planning tests verifying filter/assignment extraction

ethan-tyler · 2025-12-09T05:32:52Z

@alamb addressed your feedback and pushed the changes:

Removed DmlCapabilities entirely
Simplified planner to try-call pattern with .context() error wrapping
Added planning tests (dml_planning.rs) - 7 tests with mock providers that capture filters/assignments

Clippy is clean locally. Ready for another look when you get a chance.

Update expected error messages in sqllogictests to match new error format from .context() wrapping in physical planner. The error messages now include context about the DML operation being performed. Old format: physical_plan_error Error during planning: Table 't1' does not support DELETE operations New format: physical_plan_error 01)DELETE operation on table 't1' 02)caused by 03)This feature is not implemented: DELETE not supported for Base table

adriangb · 2025-12-09T15:29:51Z

@ethan-tyler could you update the PR description since DmlCapabilities is gone?

adriangb · 2025-12-09T15:32:25Z

datafusion/core/tests/custom_sources_cases/dml_planning.rs

+
+//! Tests for DELETE and UPDATE planning to verify filter and assignment extraction.
+
+use std::any::Any;


How hard would it be to integrate this (or a different implementation) into our in-memory tables? It would be very cool to see an SLT test showing it working.

Good timing - I had this sitting in a separate branch to keep the API PR small. Merged it in now. Adds delete_from() and update() to MemTable as a working example. Two new SLT files:

MemTable.delete_from() / MemTable.update() - reference implementation

dml_delete.slt (202 lines, 10 tests) - DELETE operations

dml_update.slt (259 lines, 12 tests) - UPDATE operations

Covers the basics plus NULL predicate handling (important for SQL correctness). The existing delete.slt/update.slt now expect success since MemTable supports these operations.

ethan-tyler · 2025-12-09T17:58:46Z

@ethan-tyler could you update the PR description since DmlCapabilities is gone?

Thanks for the catch - updated!

adriangb

Some small nits otherwise looks good to me!

adriangb · 2025-12-18T03:14:36Z

datafusion/catalog/src/memory/table.rs

+                let filter_mask = if filters.is_empty() {
+                    BooleanArray::from(vec![true; batch.num_rows()])
+                } else {
+                    let mut combined_mask: Option<BooleanArray> = None;
+
+                    for filter_expr in &filters {
+                        let physical_expr = create_physical_expr(
+                            filter_expr,
+                            &df_schema,
+                            state.execution_props(),
+                        )?;
+
+                        let result = physical_expr.evaluate(batch)?;
+                        let array = result.into_array(batch.num_rows())?;
+                        let bool_array = array
+                            .as_any()
+                            .downcast_ref::<BooleanArray>()
+                            .ok_or_else(|| {
+                                datafusion_common::DataFusionError::Internal(
+                                    "Filter did not evaluate to boolean".to_string(),
+                                )
+                            })?
+                            .clone();
+
+                        combined_mask = Some(match combined_mask {
+                            Some(existing) => and(&existing, &bool_array)?,
+                            None => bool_array,
+                        });
+                    }
+
+                    combined_mask.unwrap_or_else(|| {
+                        BooleanArray::from(vec![true; batch.num_rows()])
+                    })
+                };


Can this be factored into common code evaluate_filters_to_mask or something?

I extracted - it returns Option now so callers can handle the empty-filter case directly.

adriangb · 2025-12-18T03:15:28Z

datafusion/catalog/src/memory/table.rs

+                }
+
+                let filter_mask = if filters.is_empty() {
+                    BooleanArray::from(vec![true; batch.num_rows()])


Can't we just exit early here?

Good call - now returns None for empty filters instead of allocating an all-true mask.

adriangb · 2025-12-18T03:17:03Z

datafusion/catalog/src/memory/table.rs

+                        let physical_expr = create_physical_expr(
+                            value_expr,
+                            &df_schema,
+                            state.execution_props(),
+                        )?;


Can we create these upfront e.g. assignment_map.map(...) -> HashMap<String, Arc<dyn PhysicalExpr>>?

Done. Used String keys to avoid borrow-across-await.

# Conflicts: # datafusion/catalog/src/memory/table.rs

- Use evaluate_selection() instead of evaluate() for UPDATE assignments to avoid evaluating expressions on rows excluded by WHERE clause. This prevents errors like divide-by-zero on non-updated rows. - Extract evaluate_filters_to_mask() helper shared by delete_from/update - Hoist physical expression creation outside batch loop for efficiency - Simplify strip_column_qualifiers() signature - Add regression test for divide-by-zero edge case

alamb · 2025-12-30T12:38:48Z

Thanks @ethan-tyler and @adriangb
I just merged up from main to rerun the tests

Are there any concerns about merging this PR?

adriangb · 2025-12-30T12:54:56Z

I think it's good to go!

ethan-tyler · 2025-12-30T15:25:39Z

Thanks @ethan-tyler and @adriangb

I just merged up from main to rerun the tests

Are there any concerns about merging this PR?

No additional concerns! Thank you so much @alamb @adriangb for helping on this one!

alamb · 2025-12-31T00:33:24Z

🎉

github-actions bot added logical-expr Logical plan and expressions core Core DataFusion crate catalog Related to the catalog crate labels Dec 7, 2025

ethan-tyler added 5 commits December 7, 2025 13:26

test: Add unit tests for DmlCapabilities

0e9e691

fix: Use assert_eq! instead of assert! for const values

695716c

Clippy warns about assertions on constants being optimized away. Changed to assert_eq! for explicit value comparison.

test: Update existing delete.slt and update.slt for MemTable DML

5b35c4c

Now that MemTable supports DELETE and UPDATE, the existing EXPLAIN tests produce physical plans instead of "not implemented" errors. Updated expected output to reflect the new DmlResultExec execution.

ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from 7505171 to 695716c Compare December 7, 2025 18:29

ethan-tyler added 2 commits December 7, 2025 16:47

Merge pull request #1 from ethan-tyler/feat/tableprovider-dml-hooks

01d7e0e

Feat/tableprovider dml hooks

Merge pull request #2 from ethan-tyler/feat/memtable-dml-impl

a59d9a6

Feat/memtable dml impl

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Dec 8, 2025

ethan-tyler added 4 commits December 7, 2025 19:45

test: Add unit tests for DmlCapabilities

04fd988

fix: Use assert_eq! instead of assert! for const values

30d98c5

Clippy warns about assertions on constants being optimized away. Changed to assert_eq! for explicit value comparison.

fix: Use assert!() for bool assertions and update expected DML errors

b038d04

- Change assert_eq!(x, true/false) to assert!(x)/assert!(!x) per clippy - Update delete.slt and update.slt expected error messages to match new physical planner behavior for unsupported DML operations

ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from 22c7273 to b038d04 Compare December 8, 2025 00:45

ethan-tyler added 2 commits December 7, 2025 19:47

test: Update existing delete.slt and update.slt for MemTable DML

4adc590

Now that MemTable supports DELETE and UPDATE, the existing EXPLAIN tests produce physical plans instead of "not implemented" errors. Updated expected output to reflect the new DmlResultExec execution.

alamb reviewed Dec 8, 2025

View reviewed changes

alamb mentioned this pull request Dec 8, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-12-08 #19210

Closed

40 tasks

ethan-tyler added 2 commits December 8, 2025 23:40

refactor: Remove DmlCapabilities, simplify DELETE/UPDATE API

3e635b3

- Remove DmlCapabilities struct and dml_capabilities() trait method - Simplify physical planner to try-call pattern with .context() errors - Add planning tests verifying filter/assignment extraction

refactor: Remove DmlCapabilities, simplify DELETE/UPDATE API

a713047

- Remove DmlCapabilities struct and dml_capabilities() trait method - Simplify physical planner to try-call pattern with .context() errors - Add planning tests verifying filter/assignment extraction

ethan-tyler added 2 commits December 9, 2025 07:08

Merge branch 'main' into feat/tableprovider-dml-hooks

14d6075

adriangb reviewed Dec 9, 2025

View reviewed changes

ethan-tyler added 4 commits December 9, 2025 12:32

Merge feat/memtable-dml-impl: Add MemTable DML implementation and tests

c67dc67

Merge main into feat/tableprovider-dml-hooks

9a206fa

fix: Remove stale DmlCapabilities exports from merge

bee5f1e

Merge branch 'main' into feat/tableprovider-dml-hooks

8cf796c

ethan-tyler mentioned this pull request Dec 11, 2025

feat(datafusion): Full DML support for tables with evolved partition specs apache/iceberg-rust#1923

Closed

ethan-tyler requested review from adriangb and alamb December 12, 2025 14:13

adriangb approved these changes Dec 18, 2025

View reviewed changes

ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from 8a7733f to d4da744 Compare December 19, 2025 03:41

ethan-tyler added 2 commits December 18, 2025 22:46

Merge branch 'main' into feat/tableprovider-dml-hooks

cd472f1

# Conflicts: # datafusion/catalog/src/memory/table.rs

ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from d4da744 to 94a2b68 Compare December 19, 2025 03:48

Merge branch 'main' into feat/tableprovider-dml-hooks

f75188d

alamb changed the title ~~feat: Add DELETE/UPDATE hooks to TableProvider trait~~ feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation Dec 30, 2025

alamb mentioned this pull request Dec 30, 2025

Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) #18566

Open

41 tasks

adriangb added this pull request to the merge queue Dec 30, 2025

Merged via the queue into apache:main with commit d13d891 Dec 30, 2025
33 checks passed

ethan-tyler mentioned this pull request Jan 3, 2026

[EPIC] Complete DML Support (MERGE, INSERT OVERWRITE, TRUNCATE) #19617

Open


		//! Tests for DELETE and UPDATE planning to verify filter and assignment extraction.

		use std::any::Any;

feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation #19142

feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation #19142

Uh oh!

Conversation

ethan-tyler commented Dec 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

ethan-tyler commented Dec 7, 2025

Uh oh!

ethan-tyler commented Dec 7, 2025

Uh oh!

ethan-tyler commented Dec 8, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethan-tyler commented Dec 9, 2025

Uh oh!

ethan-tyler commented Dec 9, 2025

Uh oh!

adriangb commented Dec 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethan-tyler commented Dec 9, 2025

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Dec 30, 2025

Uh oh!

adriangb commented Dec 30, 2025

Uh oh!

ethan-tyler commented Dec 30, 2025

Uh oh!

Uh oh!

alamb commented Dec 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ethan-tyler commented Dec 7, 2025 •

edited

Loading