Skip to content

Handle Delta StructType, ArrayType and MapType#16884

Merged
abhishekrb19 merged 8 commits intoapache:masterfrom
abhishekrb19:delta_complex_types_impl
Aug 13, 2024
Merged

Handle Delta StructType, ArrayType and MapType#16884
abhishekrb19 merged 8 commits intoapache:masterfrom
abhishekrb19:delta_complex_types_impl

Conversation

@abhishekrb19
Copy link
Copy Markdown
Contributor

@abhishekrb19 abhishekrb19 commented Aug 13, 2024

Fixes #16782.

Changes:

  • Handle the following Delta complex types:
    a. StructType as JSON
    b. ArrayType as Java list
    c. MapType as Java map

  • Generate and add a new Delta table complex-types-table that contains the above complex types for testing.

  • Update the tests to include a parameterized test with complex-types-table, with the expectations defined in ComplexTypesDeltaTable.java.

Screenshot with with Delta table containing complex types:

CleanShot 2024-08-12 at 18 58 00@2x

Release note

Fixes #16782 by adding support for Delta structs, arrays and maps in the delta input source.

This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

druid-deltalake-extensions support for StructType

3 participants