Support Delta lake relation

[sezruby]

What is the context for this pull request?

• Tracking Issue: n/a
• Parent Issue: Hyperspace for Delta Lake #148
• Dependencies: Pluggable source provider #227

What changes were proposed in this pull request?

Support Delta lake relation for "immutable" dataset.
This PR covers index creation/refresh(full mode)/application using FilterIndexRule and JoinIndexRule.

As we can rely on Delta Lake for data change and management, the version info and path of Delta Lake relation can be used in FileBasedSignatureProvider - instead of hash computation of all source files:

...
 case LogicalRelation(
          HadoopFsRelation(location: PartitioningAwareFileIndex, _, _, _, _, _),
          _,
          _,
          _) =>
        fingerprint ++= location.allFiles.foldLeft("")(
          (accumulate: String, fileStatus: FileStatus) =>
            HashingUtils.md5Hex(accumulate + getFingerprint(fileStatus)))
 case LogicalRelation(
          HadoopFsRelation(location: TahoeLogFileIndex, _, _, _, _, _),
          _,
          _,
          _) =>
        fingerprint ++= location.tableVersion + location.path.toString
...

Does this PR introduce any user-facing change?

Yes, a user can create/refresh/apply indexes on Delta Lake relation.

How was this patch tested?

Unit test

[imback82]

I think we should do "provided" since few cloud providers come with delta JARs built in.

[imback82]

Also, let's not worry about 0.7.0 since it is for Spark 3.0

[imback82]

Can we use matchingFiles or not?

Can you add comment why Seq() is fine for partition/data filters?

[sezruby]

matchingFiles seems to be used when there are additional data filters or partition filters. So I think we don't need to use it for now.

And I added partitionFilter.

[imback82]

Question: Is it possible that the location is TahoeBatchFileIndex?

[sezruby]

It's for version range and used in merge/update command internally. I think it would be better not to allow index creation with it to avoid complexity. WDYT? @imback82

[imback82]

Btw, I am working on the extension model so that you can plug in new source formats. I will have the PR up soon so that we can test it using Delta first.

[apoorvedave1]

the approach looks fine to me. Thanks @sezruby

[sezruby]

This PR will be delivered by #265.

[sezruby]

Closed by #265