Merge incremental mode append and delete actions into index refresh

[pirz]

Describe the issue

Adding support for enforcing append and delete during index refresh (i.e. "incremental" mode) is being done thru two parallel PRs:

• Enforce delete during refresh - Issue Add support for deletes in RefreshIndex for incremental mode #105
• Implement incremental refresh indexes of append-only data - Issue Implement smart refresh indexes of append-only data  #112

In order to make these PRs move forward with least conflicts, we decided to follow below steps:

• Step 1. Do not make changes in refresh API and protect each action under a separate temporary Spark configuration flag. To keep current behavior of refresh intact, the flag for each of the actions is disabled by default.
• Step 2. Let the PRs be reviewed, tested and merged to master.
• Step 3. Refactor refresh index code and merge above actions as a subsequent step.

This issue is created to track the 3rd step above.

Expected behavior

Refresh index code in the smart mode should support both append and delete. This means once refreshIndex(mode = "smart") is called, Hyperspace should detect both additional and deleted source files and updates the index as expected for the smart mode.

Environment

All.

[pirz]

CC @imback82 and @sezruby - FYI & please leave comments if anything needs to be changed

[pirz]

Closed by PR #187