[BEAM-191] Remove WindowedValue.valueInEmptyWindows #316
Conversation
A value in empty windows expands to no values, so it can be dropped at any time, perhaps unintentionally. This has bitten runner authors, including Spark & Dataflow. While creating such a thing in memory is not automatically problematic, it is also not really useful. So this change removes it.
|
R: @bjchambers or @amitsela or really anyone. References are this dev thread which led to #179 fixing BEAM-189. It has also be re-filed as BEAM-230. |
|
Jenkins test is failing IT test, is it running with the DataflowPipelineRunner - meaning as a Dataflow service ? If so, I think you miss an update there.. (ed3676eda3f9031e): java.lang.NoSuchMethodError: org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows(Ljava/lang/Object;)Lorg/apache/beam/sdk/util/WindowedValue; ...... (9349dfed44f1940a): Workflow failed. Causes: (b529cf23ada96ef9): S04:WriteCounts/DataflowPipelineRunner.BatchTextIOWrite/Write/DataflowPipelineRunner.BatchWrite/View.AsSingleton/BatchViewAsSingleton/DataflowPipelineRunner.GroupByWindowHashAsKeyAndWindowAsSortKey/DataflowPipelineRunner.GroupByKeyAndSortValuesOnly/Read+WriteCounts/DataflowPipelineRunner.BatchTextIOWrite/Write/DataflowPipelineRunner.BatchWrite/View.AsSingleton/BatchViewAsSingleton/ParDo(IsmRecordForSingularValuePerWindow) failed. |
|
It seems that ValueInEmptyWindows is unused now, if we don't want to remove it yet, should we |
|
Ah, yes, good catch. I will wipe out |
|
Getting in just under 1-month-idle.. Any update here? |
|
Yes, there are some prerequisites that need to be addressed first. |
|
I am going to close this while I fix the bug in the runner. |
…uments (apache#316) * Began refactoring away from GetDocument proto which contains no `read_time` field and in general is a shortcut around `BatchGetDocuments` * Correctly instantiate snapshots for missing documents * Removed stale NotFound exception * Removed unnecessary empty list check * Linting fix * Expanded batch_get change to async classes * Updated variable name * Added get_batch to async classes * Improved consumption of async generators * Fixed test coverage * Fixed broken mock in test * Linting * Reverted the move of AsyncIter Co-authored-by: Craig Labenz <craiglabenz@google.com>
3 participants
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.
A value in empty windows expands to no values, so it can be dropped at
any time, perhaps unintentionally. This has bitten runner authors, including
Spark & Dataflow.
While creating such a thing in memory is not automatically problematic, it
is also not really useful. So this change removes it.