[SPARK-30336][SQL][SS] Move Kafka consumer-related classes to its own package #26991
Conversation
|
|
||
| import java.{util => ju} | ||
|
|
||
| import scala.collection.mutable.ArrayBuffer |
There was a problem hiding this comment.
I've also cleaned up unused imports and remove unnecessary line (two empty lines between the last import statement and class definition)
| @@ -218,4 +218,3 @@ private[kafka010] object InternalKafkaConsumerPool { | |||
| } | |||
| } | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
There were two empty lines which can be just one empty line.
|
Test build #115666 has finished for PR 26991 at commit
|
|
Test build #115667 has finished for PR 26991 at commit
|
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
Looks OK as a cleanup, pending tests
|
retest this, please |
|
Test build #115751 has finished for PR 26991 at commit
|
|
Test build #115793 has finished for PR 26991 at commit
|
|
cc. @srowen Gently ping. |
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
If tests pass and it's just a refactoring, OK.
I guess you also made everything private to .consumer that you can?
Yes, the KafkaDataConsumer itself, and acquire and release are open to other package (kafka010). Others are private to consumer. |
|
Merged to master |
|
Thamks for reviewing and merging! Happy new year! |
4 participants
What changes were proposed in this pull request?
There're too many classes placed in a single package "org.apache.spark.sql.kafka010" which classes can be grouped by purpose.
As a part of change in SPARK-21869 (#26845), we moved out producer related classes to "org.apache.spark.sql.kafka010.producer" and only expose necessary classes/methods to the outside of package. This patch applies the same to consumer related classes.
Why are the changes needed?
Described above.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing UTs.