Use FileFormat-based data source instead of HadoopRDD for reads #289
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JoshRosen
commented
Oct 25, 2016
| sparkSession: SparkSession, | ||
| options: Map[String, String], | ||
| path: Path): Boolean = { | ||
| // Redshift unload files are not splittable because records containing newline characters may |
Contributor
Author
There was a problem hiding this comment.
Per discussion with @mengxr, it sounds like the existing InputFormat already does support splitting, so this comment is incorrect. I'll go ahead and update this.
Current coverage is 88.25% (diff: 89.70%)@@ master #289 diff @@
==========================================
Files 12 15 +3
Lines 702 732 +30
Methods 568 591 +23
Messages 0 0
Branches 134 141 +7
==========================================
+ Hits 622 646 +24
- Misses 80 86 +6
Partials 0 0
|
liancheng
reviewed
Oct 25, 2016
| * An adaptor from a Hadoop [[RecordReader]] to an [[Iterator]] over the values returned. | ||
| * | ||
| * Note that this returns [[Object]]s instead of [[InternalRow]] because we rely on erasure to pass | ||
| * column batches by pretending they are rows. |
Contributor
There was a problem hiding this comment.
This paragraph comes from Spark source code and doesn't really apply here. It confused me for a while.
Contributor
Author
There was a problem hiding this comment.
Yeah, this is confusing to me as well, especially since the return type is generic. I'll just remove this comment.
Contributor
|
LGTM! |
Contributor
Author
|
Great, merging to master! |
JoshRosen
added a commit
that referenced
this pull request
Oct 25, 2016
This patch refactors this library's read path to use a Spark 2.0's `FileFormat`-based data source to read unloaded Redshift output from S3. This approach has a few advantages over using our existing `HadoopRDD`-based approach: - It will benefit from performance improvements in `FileScanRDD` and `HadoopFsRelation`, including automatic coalescing. - We don't have to create a separate RDD per partition and union them together, making the RDD DAG smaller. The bulk of the diff are helper classes copied from Spark and `spark-avro` and inlined here for API compatibility / stability purposes. Some of the new classes implemented here are likely to become incompatible with new releases of Spark, but note that `spark-avro` itself relies on similar unstable / experimental APIs and thus this library is already vulnerable to changes to those APIs (in other words, this change is not making our compatibility story significantly worse). Author: Josh Rosen <joshrosen@databricks.com> Author: Josh Rosen <rosenville@gmail.com> Closes #289 from JoshRosen/use-fileformat-for-reads.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch refactors this library's read path to use a Spark 2.0's
FileFormat-based data source to read unloaded Redshift output from S3. This approach has a few advantages over using our existingHadoopRDD-based approach:FileScanRDDandHadoopFsRelation, including automatic coalescing.The bulk of the diff are helper classes copied from Spark and
spark-avroand inlined here for API compatibility / stability purposes. Some of the new classes implemented here are likely to become incompatible with new releases of Spark, but note thatspark-avroitself relies on similar unstable / experimental APIs and thus this library is already vulnerable to changes to those APIs (in other words, this change is not making our compatibility story significantly worse).