Add RefreshIncrementalAction to support index creation on newly appended data

[apoorvedave1]

Implement incremental indexing support for append-only data.

NOTE TO REVIEWERS

Uber Issue: Please follow: #136 for complete details.

This PR depends on PR #142 (Delete support on source data) and #162 (Merge support for directory objects) for major changes. Until the other PR goes in, This will be kept as WIP. It can still be viewed for general idea of how the classes will evolve.

Most files in the dependency PRs can be ignored in this PR, if the reviewer is familiar with them. The new, review-able files with new functionality are as follows:
for Functionality: RefreshIncremental.scala,
for Tests: IndexManagerTests.scala

What this PR does

This feature allows hyperspace to create indexes on newly arrived data. If the user appends new data to existing, pre-indexed data, they can use refresh api to generate indexes only on the additional data.
This index creation will be faster than full refresh because it works only on additional data. This is different for a full refresh where the index is built from scratch on full data.

Algorithm Outline:

1. identify newly added data files
2. create new index version on these files
3. update metadata to reflect the latest snapshot of index. The latest snapshot points to the complete snapshot of the index, including all the old and the newly created index files. The same is true for data files as well.

What changes were proposed in this pull request?

A new RefreshIncremental.scala action class which is built based on RefreshActionBase class. Reviewers can start from this class to understand what data is being indexed, and how the new metadata is being generated to reflect the latest truth of the index.

Why are the changes needed?

To support incremental indexing on just the unindexed data

Does this PR introduce any user-facing change?

Yes, this feature introduces a support for creating (or 'updating') index on newly added data, by creating index only on the new data.

How to enable this feature

This feature is currently hidden behind a flag "spark.hyperspace.index.refresh.append.enabled" which defaults to false. It will be later on supported as the api `refreshIndex(indexName, mode="quick") along with support for other features (e.g. delete) within that api. Please follow #136 for complete details.

How was this patch tested?

[rapoth]

@apoorvedave1 Can you look at the suggestions I made for #142 as far as the section Are there any user-facing changes? is concerned and update your PR description as appropriate?

[imback82]

LGTM, thanks @apoorvedave1!

[sezruby]

LGTM, thanks @apoorvedave1!