Support for reading Delta Lake table snapshots

[abhishekrb19]

Problem

Currently, the delta input source only supports reading from the latest snapshot of the given Delta Lake table. This is a known documented limitation.

Description

Add support for reading Delta snapshot. By default, the Druid-Delta connector reads the latest snapshot of the Delta table in order to preserve compatibility. Users can specify a snapshotVersion to ingest change data events from Delta tables into Druid.

In the future, we can also add support for time-based snapshot reads. The Delta API to read time-based snapshots is not clear currently.

Examples:

A. To ingest snapshot version 3 of the Delta table:

REPLACE INTO "snapshot_delta_table" OVERWRITE ALL
SELECT 
"id",
"map_info"
FROM TABLE(
  EXTERN(
    '{
      "type": "delta",
      "tablePath": "/Users/abhishek/projects/snapshot-table",
      "snapshotVersion": 3
    }',
    '{"type":"json"}'
  )
) EXTEND ("id" BIGINT, "map_info" TYPE('COMPLEX<json>'))
PARTITIONED BY ALL

B. To ingest a latest snapshot of the Delta table:

REPLACE INTO "snapshot_delta_table" OVERWRITE ALL
SELECT 
"id",
"map_info"
FROM TABLE(
  EXTERN(
    '{
      "type": "delta",
      "tablePath": "/Users/abhishek/projects/snapshot-table",
    }',
    '{"type":"json"}'
  )
) EXTEND ("id" BIGINT, "map_info" TYPE('COMPLEX<json>'))
PARTITIONED BY ALL

Release note

Users can optionally specify a snapshotVersion in the delta input source payload to ingest versioned snapshots from a Delta Lake table. By default, Druid ingests the latest snapshot of the Delta Lake table when snapshotVersion is not specified.

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.