WIP: ARROW-1796: [Python/Parquet] RowGroup filtering on file level #2623
Conversation
|
Just as a heads-up. This is missing some more granular tests and the exact filtering converts to Pandas and back again. Changing that to work on pure Arrow tables will be a lot more work. I will separate out some things into smaller pull requests. |
|
cc @rgruener I think you were one of the interested people in this topic. |
rgruener
left a comment
rgruener
left a comment
There was a problem hiding this comment.
Changes look good, thanks for this!
| if isinstance(filters[0][0], six.string_types): | ||
| # We have encountered the situation where we have one nesting level too few: | ||
| # We have [(,,), ..] instead of [[(,,), ..]] | ||
| filters = [filters] |
There was a problem hiding this comment.
This automatic fix isn't my favorite though I see that it is necessary to not have a braking change in the api for ParquetDataset (with the filters argument). Perhaps though it would be better to throw an error here and have this fix in that specific case instead of allowing a wrong nesting level in all cases.
| inner_indexer = (ser < value) & inner_indexer | ||
| elif op == ">": | ||
| inner_indexer = (ser > value) & inner_indexer | ||
| elif op == "in": |
| return min_value < val | ||
| elif op == ">": | ||
| return max_value > val | ||
| elif op == "in": |
| self.validate_schemas() | ||
|
|
||
| if filters: | ||
| if filters is not None: |
There was a problem hiding this comment.
filters can now be either List[Tuple] or List[List[Tuple]] and can either filter on partitions or row groups depending on if the dataset is partitioned. Either way the current state should be mentioned in the docstring (unless that will be worked out before this PR is merged)
|
@xhochy is this still WIP? Anything I can help with? |
|
@emkornfield I'm not actively working on this at the moment. Feel free to pick this up or just to integrate parts of the PR into arrow. |
|
Closing this for now. I'll refer to this PR in the course of implementing Scanner logic in the Datasets project |
4 participants
No description provided.