TAJO-1952: Implement PartitionFileFragment

[blrunner]

This patch contains following modifications:

• Remove partition paths from PartitionedTableScanNode
• Implement PartitionedFileFragment
• Separate a method for pruning partition paths from PartitionedTableRewriter to PartitionedTableUtil
• Build a type by using partition name which contains partition keys and values

[hyunsik]

I'm going to give a suggestion. We need to keep each method for one purpose if possible. That approach would keep logic simpler.

[hyunsik]

Could you rebase it against the latest revision? I'd like to try the patch on my machine.

[blrunner]

@hyunsik

Thank you for your review. I've just rebased it against the latest version.

[hyunsik]

In my opionion, partitionKeys would be more proper because the attribute includes concatenated partition keys.

[blrunner]

@hyunsik

Thank your for your review. I've just reflected your comments.

[blrunner]

I updated the patch as following:

• Renamed PartitionedFileFragment to PartitionFileFragment
• Recovered codes for pruning partition in PartitionedTableRewrtier
• Recovered partition paths of PartitionedTableScanNode and added partition keys to PartitionedTableScanNode
• Implemented PartitionContent which contains partition paths, partition keys, partition volume

For the reference, I've tried to maintain existing codes as far as possible.

[hyunsik]

unused constructor

[blrunner]

@hyunsik

Updated the patch as following:

• Remove unused packages
• Remove PartitionedTableScanNode of PlanProto
• Remove unnecessary PartitionedTableScanNode usages of Repartitioner and PhysicalPlannerImpl

[blrunner]

Added unit test cases for PartitionedTableRewriter

[hyunsik]

It can be simplified with lambda.

[blrunner]

@hyunsik

Thank you for your detailed review. I reflected your comments.

[blrunner]

@hyunsik

I changed PartitionFileFragment by extending FileFragment because there are some codes for only FileFragment. For example, you can see cast codes as following:

  public FileScanner(Configuration conf, final Schema schema, final TableMeta meta, final Fragment fragment) {
    this.conf = conf;
    this.meta = meta;
    this.schema = schema;
    this.fragment = (FileFragment)fragment;
    this.tableStats = new TableStats();
    this.columnNum = this.schema.size();
  }

[blrunner]

Evaluated patch testing before and after PartitionFileFragment implementation as following:

• Dataset : TPCH-100G
• Tajo Cluster : 1 Master, 6 Workers
• Queries: Q1, Q3, Q5, Q6, Q7, Q8, Q9, Q10
• Tajo version for before PartitionFileFragment implementation
  • 0.11.1-SNAPSHOT
• Tajo version for after PartitionFileFragment implementation:
  • 0.12.0-SNAPSHOT (partitions exist on catalog)
  • 0.12.0-SNAPSHOT (partitions doesn't exist on catalog)

The results were same as following:

• The number and order of execution blocks
• The number of tasks in a each execution block
• The number of rows in a result
• All tuples in a result (excluded a few floating point value)

[blrunner]

The Travis CI Build seems like a fail by another reason. Here is the pre-commit report by lasted patch as follows. https://builds.apache.org/job/PreCommit-TAJO-Build/900//console

[blrunner]

@hyunsik

Is there a problem that needs to be fixed?

[blrunner]

@hyunsik

I updated this PR as following:

• Remove the list of partition paths in PartitionedTableScanNode. It just contains the path of table root.
• PartitionedTableRewriter just rewrite ScanNode to PartitionedTableScanNode in BaseLogicalPlanRewriteRuleProvider.
• Actual partition pruning will be executed in Repartitioner by PartitionedTableRewriter.