Support realtime view in Spark datasource

[prazanna]

Need to support hoodie merge on read in HoodieReadClient

[vinothchandar]

Early thoughts.

Implement a Spark File Format

• HoodieRealtimeFileFormat modelled after ParquetFileFormat inside Spark
• Can wrap org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat & another piece which merges output from log as Iterator[Row]
• Use HadoopFsRelation to pass in the file format if needed, with schema & everything

Reading output from the IPF RecordReader

• org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport inside Spark helps convert Parquet records to Row.
• However, current IPFs HoodieInputFormat & HoodieRealtimeInputFormat both extend from MapredParquetInputFormat, which seems to hardcode ArrayWritable as the written value. hence the core merge/compaction logic needs to have templatized types to support this.

public class MapredParquetInputFormat extends FileInputFormat<Void, ArrayWritable> {
    private static final Log LOG = LogFactory.getLog(MapredParquetInputFormat.class);
    private final ParquetInputFormat<ArrayWritable> realInput;

• We won't be calling the RecordReader level apis directly at all.

Filtering & listing such that only right splits are considered.

• TBD. Guess, is forcing convertMetastoreParquet=false would get Spark to call ipf.getSplits() which does the job and later only valid splits are passed into the closure returned from FileFormat
• Per code in DataSource.scala pruning paths in the path option before creating the relation would work too

case (format: FileFormat, _) =>
        val allPaths = caseInsensitiveOptions.get("path") ++ paths
        val hadoopConf = sparkSession.sessionState.newHadoopConf()

[vinothchandar]

spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala contains code, that will transform a given Writable into Row in method fillObject

[vinothchandar]

Moved to https://issues.apache.org/jira/browse/HUDI-69