Skip to content

[VL][Delta] Add native deletion vector scan foundation#11900

Draft
malinjawi wants to merge 30 commits intoapache:mainfrom
malinjawi:delta-dv-read-foundation
Draft

[VL][Delta] Add native deletion vector scan foundation#11900
malinjawi wants to merge 30 commits intoapache:mainfrom
malinjawi:delta-dv-read-foundation

Conversation

@malinjawi
Copy link
Copy Markdown
Contributor

@malinjawi malinjawi commented Apr 9, 2026 •
edited
Loading

Issue: #11901

What changes are proposed in this pull request?

This PR adds the native Delta deletion vector scan foundation for the Velox backend in Gluten.

The architecture is based of the design of Delta Lake's: Deletion Vectors High Level Design.

The scope is intentionally limited to the read path. It introduces the JVM and native plumbing required to read Delta tables with deletion vectors natively, while keeping DML and DV write-path work out of scope for a follow-up PR.

The main changes are:

  • Add JVM-side Delta scan preparation and metadata normalization for DV-aware scans.
  • Add native Delta scan/read components under cpp/velox/compute/delta, including:
    • Delta connector and datasource plumbing
    • Delta split representation
    • deletion vector reader
    • split reader integration
    • UUID / Base85 utilities for Delta DV handling
  • Add explicit Roaring dependency wiring and a local RoaringBitmapArray helper needed by the native DV read path and tests.
  • Wire the Delta read path into the Velox backend and scan execution flow.
  • Add focused native tests for:
    • Delta connector
    • deletion vector reader
    • split handling
    • UUID utilities

This PR does not include:

  • row-index finder integration
  • DELETE / UPDATE / MERGE DV write-path changes
  • compaction / utility command support

How was this patch tested?

  • Added native tests under cpp/velox/compute/delta/tests:
    • DeltaConnectorTest
    • DeltaDeletionVectorReaderTest
    • DeltaSplitTest
    • DeltaUuidUtilsTest
  • Locally validated that the new Delta read sources are integrated into the native velox target and compile through the new Delta translation units.
  • Locally validated explicit Roaring discovery and removed unresolved CRoaring symbols after wiring the dependency.

Was this patch authored or co-authored using generative AI tooling?

Co-authored: IBM Bob

Mohammad Linjawi added 2 commits April 9, 2026 14:47
[VL][Delta] Add native DV read-only infrastructure
069a1d3 
Implements complete read-only Deletion Vector support for Delta Lake:
- C++ DV reader and row index finder (14 files)
- JNI bindings for native DV operations (2 files)
- Runtime integration for DV-aware scans (3 files)
- Scala preprocessing and scan preparation (3 files)
- Comprehensive test coverage (5 files)

This is the foundation for native DV MoR operations. Read-only
implementation ensures no risk of data corruption while enabling
performance testing of the native read path.

Key components:
- DeltaDeletionVectorReader: Reads DV files from storage
- DeltaRowIndexFinder: Identifies deleted rows during scans
- WholeStageResultIterator: Integrates DV filtering into result iteration
- PreprocessTableWithDVs: Prepares Delta tables with DVs for native execution

Total: 27 files (read-only DV infrastructure only)
[VL][Delta] Split DV read foundation
4aac1cd
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

7 similar comments
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@malinjawi malinjawi force-pushed the delta-dv-read-foundation branch from 7025b9b to 4fbb08e Compare April 9, 2026 18:00
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

Mohammad Linjawi and others added 16 commits April 9, 2026 21:02
[VL][Delta] Fix DV read foundation CI
1e9af3c
@sh-shamsan
[VL][CI] Adding docker image for maven cache (apache#11655)
d4ad1f4 
This patch added a docker image with maven cache for spark package/test, covering 3.3/3.4/3.5/4.0/4.1. 
A second patch will be made to enable this cache in the CI jobs
@JkSelf
[VL] Refine logs in BHJ optimization (apache#11870)
f4d90ba 
Clean up redundant logging.

fixes: apache#11863
@zhouyuan
[GLUTEN-11872][VL] Fix docker build (apache#11873)
e984dc0 
bump the actions to match apache policies, also fixed the cache image build

Signed-off-by: Yuan <yuanzhou@apache.org>
@GlutenPerfBot @baibaichen
@zhouyuan
[GLUTEN-6887][VL] Daily Update Velox Version (2026_04_01) (apache#11860)
799ed03 
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_04_01)

Upstream Velox's New Commits:
24e6ab97b by Chengcheng Jin, fix(cudf): Fix complex data type name in format conversion and add tests(Part1) (#16818)
d92b90029 by Natasha Sehgal, refactor: Propagate CastRule cost through canCoerce (#16821)
361a42252 by Rui Mo, fix(fuzzer): Reduce Spark aggregate fuzzer test pressure (#16964)
2c2fe2ab7 by root, fix: Ignore string column statistics for parquet-mr versions before 1.8.2 (#16744)
7faf27a86 by Chengcheng Jin, feat(cudf): Add the log to show detailed fallback messgae (#16900)
e603315e5 by Chang chen, feat(parquet): Add type widening support for INT and Decimal types with configurable narrowing (#16611)
1e1674dd8 by Rajeev Singh, docs: Add blog post for Adaptive per-function CPU tracking (#16945)
0c6b89d61 by Masha Basmanova, fix(build): Guard fuzzer examples subdirectory with VELOX_BUILD_TESTING (#16992)
8d6355d8d by Pratik Pugalia, build: Improve build impact comment layout (#16971)
44d561990 by Masha Basmanova, refactor: Add ConnectorRegistry class with tryGet and unregisterAll (#16977)
793f13f16 by Rajeev Singh, feat(expr-eval):Adaptive per-function CPU sampling for Velox expression evaluation (#16646)
1a4dc7a5a by Pratik Pugalia, fix: Off-by-one boundary bug in make_timestamp validation (#16944)
7f2c75c26 by Pratik Pugalia, Fix incorrect substr length in Tokenizer::matchUnquotedSubscript (#16972)
22b90045e by Masha Basmanova, docs: Add truncate markers to blog posts for cleaner listing page (#16975)

Signed-off-by: glutenperfbot <glutenperfbot@glutenproject-internal.com>

* Fix SPARK-18108: exclude partition columns from HiveTableHandle dataColumns

When Gluten creates HiveTableHandle, it was passing all columns (including
partition columns) as dataColumns. This caused Velox's convertType() to
validate partition column types against the Parquet file's physical types,
failing when they differ (e.g., LongType in file vs IntegerType from
partition inference).

Fix: build dataColumns excluding partition columns (ColumnType::kPartitionKey).
Partition column values come from the partition path, not from the file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Point Velox to PR3 branch with parquet type widening support

* Update VeloxTestSettings for Velox PR2

With OAP INT narrowing commit replaced by upstream Velox PR #15173:
- Remove 2 excludes now passing: LongType->IntegerType, LongType->DateType
- Add 2 excludes for new failures: IntegerType->ShortType (OAP removed)

Exclude 63 (net unchanged: -2 +2). Test results: 21 pass / 63 ignored.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Disable native writer for ParquetTypeWideningSuite

This suite tests the READ path only. Disable native writer so Spark's
writer produces correct V2 encodings (DELTA_BINARY_PACKED/DELTA_BYTE_ARRAY).
- Remove 10 excludes for decimal widening tests now passing

Remaining 38 excludes:
- 34: Velox native reader rejects incompatible decimal conversions
  regardless of reader config (no parquet-mr fallback)
- 4: Velox does not support DELTA_BYTE_ARRAY encoding

Test results: 46 pass / 38 ignored.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Override 33 type widening tests with expectError=true

Velox native reader always behaves like Spark's vectorized reader,
so tests that rely on parquet-mr behavior (vectorized=false) fail.
Instead of just excluding these 33 tests, add testGluten overrides
with expectError=true to verify Velox correctly rejects incompatible
conversions.

- 16 unsupported INT->Decimal conversions
- 6 decimal precision narrowing cases
- 11 decimal precision+scale narrowing/mixed cases

VeloxTestSettings: 38 excludes (parent tests) + 33 testGluten overrides
Test results: 79 pass / 38 ignored (33 excluded parent + 5 truly excluded)

* fix velox rebase

Signed-off-by: Yuan <yuanzhou@apache.org>

* ignore ut

Signed-off-by: Yuan <yuanzhou@apache.org>

* ignore more ut

Signed-off-by: Yuan <yuanzhou@apache.org>

* fix ignore api

Signed-off-by: Yuan <yuanzhou@apache.org>

* ignore failed ut

Signed-off-by: Yuan <yuanzhou@apache.org>

* fix on clickhouse tpcds queries

the testing data on clickhouse side is not upated, so revert to use the old query

Signed-off-by: Yuan <yuanzhou@apache.org>

* fix q30

* ignore ut

* fix

Signed-off-by: Yuan <yuanzhou@apache.org>

---------

Signed-off-by: glutenperfbot <glutenperfbot@glutenproject-internal.com>
Signed-off-by: Yuan <yuanzhou@apache.org>
Co-authored-by: glutenperfbot <glutenperfbot@glutenproject-internal.com>
Co-authored-by: Chang chen <changchen@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Chang chen <changchen@apache.org>
Co-authored-by: Yuan <yuanzhou@apache.org>
@wecharyu
[GLUTEN-11678][VL] Native validation should check CrossRelNode's expr…
912f781 
…ession (apache#11679)

Check the CrossRelNode's expression, fallback it if experssion is not supported.
Fix apache#11678.
@zhouyuan
[GLUTEN-11872][VL] Fix docker metadata action version (apache#11875)
c183749
@FelixYBW
[VL] Use UnboundedBlockingQueue when create threads pool (apache#11877)
81c04ad
@jiangjiangtian
[MINOR] Expose Gluten and component build information to SparkConf (a…
fff5256 
…pache#11838)
@Mariamalmesfer
[GLUTEN-1433] [VL] Add config to disable TimestampNTZ validation fall…
88c5375 
…back (apache#11720)

This PR adds a config to control fallback validation for TimestampNTZType in the Velox backend and adds a test for localtimestamp().

Currently, the validator treats TimestampNTZType as unsupported and forces the query to fall back to Spark. This makes it hard to develop and test features related to TimestampNTZ, including functions like localtimestamp().

With this change, the validation rule can be temporarily disabled during development and testing.

Related issue: apache#1433

Co-authored-by: Mariam-Almesfer <mariam.almesfer@ibm.com>
@exmy
[GLUTEN-11849][CH] Fix diff for var_samp returns NaN instead of NULL …
4d03d61 
…when effective row count < 2 (apache#11850)

Co-authored-by: xumingyong <xumingyong@bigo.sg>
@zzcclp
[GLUTEN-11838] Enable 'Eliminate two aggregate joins with attribute r…
cdf092a 
…eordered' suite (apache#11884)

After apache#9473 , there is an issue when executing suite 'Eliminate two aggregate joins with attribute reordered'.
@GlutenPerfBot
[GLUTEN-9219][VL] Update on function support docs (apache#11881)
b813208 
Signed-off-by: glutenperfbot <glutenperfbot@glutenproject-internal.com>
Co-authored-by: glutenperfbot <glutenperfbot@glutenproject-internal.com>
@philo-he
[MINOR] Remove Spark 3.2 tests (apache#11887)
037e2ef 
Since Spark 3.2 support was dropped a few months ago, the related tests can be removed now.

related: apache#11379
@JkSelf
[GLUTEN-11888][VL]Remove the synchronized lock in VeloxBroadcastBuild…
9dedf04 
…SideCache (apache#11834)
@JkSelf
[GLUTEN-11888] [VL] Parallel build hash table to improve bhj performa…
2af5396 
…nce (apache#11889)
@github-actions github-actions bot added the DOCS label Apr 9, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2026

Run Gluten Clickhouse CI on x86

zhztheplayer
zhztheplayer reviewed Apr 13, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@zhztheplayer zhztheplayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malinjawi Thanks.

A broad design question: Can you check if it's worth it to maintain native deletion vector reader vs. passing serialized deletion vector from Java to C++? In terms of the overall reading performance.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@malinjawi
Copy link
Copy Markdown
Contributor Author

malinjawi commented Apr 13, 2026

@malinjawi Thanks.

A broad design question: Can you check if it's worth it to maintain native deletion vector reader vs. passing serialized deletion vector from Java to C++? In terms of the overall reading performance.

Thanks for the design query @zhztheplayer.

I checked this with 2 benchmark layers:

  1. a focused DV-path microbenchmark
  2. a full end-to-end Spark benchmark on a Delta table with deletion vectors

I compared native path Vs. JVM-materialized serialized DV handoff path, where I setup as follows:

  • native side loads/materializes the deletion vector and applies it in the read path

  • JVM materializes the deletion vector first using Delta’s JVM path, serializes it as a portable Roaring payload, then passes it to native through Gluten’s existing split metadata path to decode/deserialize it and apply it in the read path

Here what I found:

Microbenchmark:

  • When full cost is included, the JVM-materialized handoff path performs essentially the same as the native path.
  • Across the inline and stored DV cases I tested, the observed difference stayed within normal benchmark noise, approximately -1% to +3% relative to the native path.
  • The only clearly faster result came from an optimistic consumer-only setup where native receives serialized DV bytes at zero cost, which does not reflect the real end-to-end design.

End-to-end Spark benchmark:

  • 1M rows, 125k deleted rows in the DV

    • native median: 627.81 ms
    • JVM handoff median: 596.75 ms
    • JVM handoff was approximately 4.9% faster in this run

  • 4M rows, 500k deleted rows in the DV

    • native median: 1556.15 ms
    • JVM handoff median: 1541.18 ms
    • JVM handoff was approximately 1.0% faster in this run

From my observation, the smaller run shows a modest advantage for the JVM-materialized handoff path, but the larger run narrows that gap to about 1%. Taken together, these results do not show a strong end-to-end performance advantage for replacing the native DV reader with a JVM-side serialized-DV handoff design. Therefore, the two approaches are effectively performance-neutral for this test, so JVM handoff path is not clearly better then native in these testing.

Based on that, I think keeping the native DV reader is still the right foundation for this PR.

I think we can introduce caching / avoiding repeated DV loads, rather than shifting the DV load boundary from native to JVM. Although what we could also checkout is a zero-copy JNI byte-transfer design.

What do you think @zhztheplayer ?

cc: @zhouyuan

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@malinjawi malinjawi force-pushed the delta-dv-read-foundation branch from 6c67e6d to a88dc84 Compare April 13, 2026 14:33
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@malinjawi malinjawi force-pushed the delta-dv-read-foundation branch from a88dc84 to d2d279e Compare April 13, 2026 14:50
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

@malinjawi malinjawi force-pushed the delta-dv-read-foundation branch from 1a08f72 to ea18efe Compare April 13, 2026 16:35
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 13, 2026

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhztheplayer zhztheplayer zhztheplayer left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

DATA_LAKE VELOX

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.