[BEAM-22] Apply ModelEnforcement in the InProcessPipelineRunner #129
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgroh
force-pushed
the
ippr_apply_enforcement
branch
6 times, most recently
from
bcc52f6 to
8613bda
Compare
Member
Author
|
R: @kennknowles |
| @Default.Boolean(true) | ||
| @Description( | ||
| "Controls whether the runner should ensure that all of the elements of every " | ||
| + "PCollection are not mutated after being output.") |
Member
There was a problem hiding this comment.
Sentence reads a bit oddly. There's grammar, but then there's also the fact that you only talk about output, rather than input. Even though technically everything that is input was output too (derp) it is still helpful to describe in terms of properties of the DoFn, like "checks that a DoFn does not modify its inputs, nor modify its outputs after they have been output".
This ensures that user code does not violate the model. Add a flag to control application of immutability enforcement. This flag is enabled by default.
Check per-element, to catch failures within a call to ProcessElement more quickly. Move wrapping of exceptions over the course of calls to ProcessElement to ParDoInProcessEvaluator.
tgroh
force-pushed
the
ippr_apply_enforcement
branch
from
b2a14cc to
150eac5
Compare
Member
|
LGTM. I will merge this shortly. |
tgroh
pushed a commit
to tgroh/beam
that referenced
this pull request
Apr 7, 2016
pl04351820
pushed a commit
to pl04351820/beam
that referenced
this pull request
Dec 20, 2023
minxhe
pushed a commit
to minxhe/beam
that referenced
this pull request
Nov 21, 2024
add config map to FlinkPipelineOptions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
number, if there is one.
Individual Contributor License Agreement.
This ensures that user code does not violate the model, as enforced
by ModelEnforcements.
Add a flag to control application of immutability enforcement. This flag
is enabled by default.
Also include minor touchups to ImmutabilityCheckingEnforcement.