Skip to content

[runners-flink] #29558 emit watermark before source shutdown #29563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
je-ik merged 1 commit into from
Dec 11, 2023
Merged

[runners-flink] #29558 emit watermark before source shutdown #29563

je-ik merged 1 commit into from
Dec 11, 2023

Conversation

@je-ik
Copy link
Contributor

@je-ik je-ik commented Nov 29, 2023

  • added position to FlinkBoundedSource checkpoint

Fixes #29558

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@je-ik
Copy link
Contributor Author

je-ik commented Nov 29, 2023

@becketqin I created this WIP solution which seems to fix the issue for me. Can you take a look? If the solution looks good, I'd add some more tests.

@je-ik je-ik force-pushed the 29558-emit-watermark branch from 8bf202e to d2895d4 Compare November 29, 2023 19:30
@je-ik
Copy link
Contributor Author

je-ik commented Nov 30, 2023

Run Flink ValidatesRunner

1 similar comment
@je-ik
Copy link
Contributor Author

je-ik commented Dec 1, 2023

Run Flink ValidatesRunner

@JozoVilcek
Copy link
Contributor

JozoVilcek commented Dec 7, 2023

@je-ik overall looks ok to me. One request, could you please add tests covering this functional behaviour.

@je-ik je-ik force-pushed the 29558-emit-watermark branch from d2895d4 to f4ff997 Compare December 11, 2023 09:46
@je-ik je-ik changed the title [WIP] [runners-flink] #29558 emit watermark before source shutdown [runners-flink] #29558 emit watermark before source shutdown Dec 11, 2023
@je-ik je-ik force-pushed the 29558-emit-watermark branch from f4ff997 to 97d70c5 Compare December 11, 2023 09:49
@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

@JozoVilcek Added test for the watermark emission, the checkpointing is covered by the updated test.

@github-actions github-actions bot removed the website label Dec 11, 2023
@je-ik
[runners-flink] apache#29558 emit watermark before source shutdown
75ba21f 
  * emit watermark when there is no more work in bounded source reader

  * added position to FlinkBoundedSource checkpoint
@je-ik je-ik force-pushed the 29558-emit-watermark branch from 97d70c5 to 75ba21f Compare December 11, 2023 09:52
@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Flink ValidatesRunner

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Java Flink PortableValidatesRunner Streaming

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Java Flink PortableValidatesRunner Batch

@github-actions
Copy link
Contributor

github-actions bot commented Dec 11, 2023

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @lostluck added as fallback since no labels match configuration

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

stop reviewer notifications

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

R: @JozoVilcek

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Flink ValidatesRunner

@github-actions
Copy link
Contributor

github-actions bot commented Dec 11, 2023

Stopping reviewer notifications for this pull request: requested by reviewer

@JozoVilcek
Copy link
Contributor

JozoVilcek commented Dec 11, 2023

R: @JozoVilcek

@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Flink ValidatesRunner

1 similar comment
@je-ik
Copy link
Contributor Author

je-ik commented Dec 11, 2023

Run Flink ValidatesRunner

@je-ik je-ik merged commit a3d0115 into apache:master Dec 11, 2023
@je-ik je-ik deleted the 29558-emit-watermark branch December 11, 2023 14:09
@je-ik je-ik mentioned this pull request Jan 19, 2024
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JozoVilcek JozoVilcek JozoVilcek approved these changes

Assignees

No one assigned

Labels

flink Next Action: Reviewers runners

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Impulse in FlinkRunner does not emit watermark

2 participants

@je-ik @JozoVilcek