Extend PulsarIO to support Batch Sources. This pr implements PIP-65 #7090
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
May 28, 2020 20:42
jerrypeng
approved these changes
Jun 2, 2020
1 task
merlimat
pushed a commit
to merlimat/pulsar
that referenced
this pull request
Jun 30, 2020
…pache#7090) * Extennd PulsarIO to support Batch Sources. This pr implements PIP-65 * Added license * Fix headers Co-authored-by: Sanjeev Kulkarni <sanjeevk@splunk.com>
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
…pache#7090) * Extennd PulsarIO to support Batch Sources. This pr implements PIP-65 * Added license * Fix headers Co-authored-by: Sanjeev Kulkarni <sanjeevk@splunk.com>
Anonymitaet
added
the
doc-required
momo-jun
added
doc-not-needed
Contributor
|
Refreshed the doc status label. See #14561 (comment) for more details. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(If this PR fixes a github issue, please add
Fixes #<xyz>.)Fixes #
(or if this PR is one task of a github issue, please add
Master Issue: #<xyz>to link to the master issue.)Master Issue: #
Motivation
This pr implements the ideas outlined in PIP-65.
It introduces BatchSource as a new interface for writing batch based connectors. It also introduces BatchSourceTriggerer as an interface to trigger the data collection of a BatchSource. It then provides system implementation in BatchSourceExecutor.
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
doc-required