Numeric lineage implementation

[pirz]

What is the context for this pull request?

This PR makes two major changes:

• Adds id generation for each source data file and index file.
• Changes lineage from file full path (stored as String) into a numeric value (Stored as Long).

For more details on the design and implementation, check below proposal.

• Proposal: [PROPOSAL]: Numeric id-based lineage implementation #220
• Tracking Issue: Numeric id-based lineage column #200

What changes were proposed in this pull request?

Assign unique ids to source data files and index files

When creating/refreshing an index, Hyperspace generates a unique id per file and use it to track the files.
Combination of (filePath, size, modificationTime) is used as the key to identify a unique file for id generation. Therefore, if content of an existing file changes, Hyperspace treats it as a new file and assigns a new id to it.
This id is stored in FileInfo and shows up in the index metadata:

• The id is for internal-use only. Hyperspace tracks the id values for each index and generates them automatically.
• For a given index, it is unique per file. If a given file is referred to in the source content of multiple indexes, it could have a different id in each of those indexes.

Change lineage from String to numeric id values

The data type for the lineage column is changed from String to an integral data type and instead of storing full path to source data file as lineage value, we store the id associated to that file as the lineage value.

Change create index plan for lineage

Instead of using a UDF to calculate/fill lineage values, the value of lineage is filled by doing a (broadcast) join between the index DF and a lineage DF that maps each source data file to its file id.

Here is a an example Create Index plan:

InsertIntoHadoopFsRelationCommand PATH_TO_IX_FOLDER, ..., bucket columns: [ColName], sort columns: [ColName], Parquet, ...
+- RepartitionByExpression [ColName], ...
   +- Project [...]
      +- Join Inner, (_data_path#10656 = _data_path#10652)
         :- Project [...]
         :  +- Relation[..., _data_path]
         +- ResolvedHint (broadcast)
            +- LocalRelation [_data_path#10652, _data_file_id#10653L]

Performance evaluation

Robustness and performance of above changes is evaluated on a workload consisting of 37 indexes of various sizes on data generated via TPCDS dsdgen for SF=1000GB and Numeric lineage with Join showed 18% to 23% of performance improvement (depending on source data file physical layout) and 5X less storage bloat when creating indexes comparing to storing full file path as lineage values.

Does this PR introduce any user-facing change?

Yes, it affects index metadata and adds new fields to it.

How was this patch tested?

Existing test cases are modified to adopt the change and are used to verify it.

[imback82]

Did one round of review. Curious to see how @sezruby's comment will change.

[pirz]

Did one round of review. Curious to see how @sezruby's comment will change.

@imback82 I am not sure I get this part. Can you plz explain which comment you are referring to and what do you expect? thnx

[imback82]

Overall approach seems fine to me.

[imback82]

Not related to this PR, but question: @sezruby can we not use FileInfo here (so that we can remove UNKNOWN_FILE_ID)?

[pirz]

@imback82 FileInfo now has id and id is meaningful in the context of a given index. At this point, we dont really know which index is gonna be picked, if any. So id is really unknown here.

[imback82]

Can't isHybridScanCandidate updated to take in Seq[FileStatus] instead? Seems like we need to change only val commonCnt = inputSourceFiles.count(entry.sourceFileInfoSet.contains) to work with FileStatus? (btw, we don't need to address this in this PR)

[sezruby]

Yes it's not necessarily FileInfo and also we can utilize tag feature to keep appended and deleted file info in getCandidateIndexes. I'll create an issue for this. Thanks

[imback82]

Thanks!

[imback82]

@sezruby Any other comments? I plan to do a release on 11/17 PST. Thanks!

[sezruby]

LGTM Thanks!

[sezruby]

Yes it's not necessarily FileInfo and also we can utilize tag feature to keep appended and deleted file info in getCandidateIndexes. I'll create an issue for this. Thanks

[imback82]

LGTM (a couple minor comments), thanks @pirz (and @sezruby for review)!

[imback82]

Thanks!

[imback82]

This class has many logics to validate and I think we need unit tests for this class. I will create an issue to follow up.

[imback82]

Created #258