Skip to content

Conversation

@seungjin-an
Copy link

This is a cherry pick from OSS

Samza runner support for non-unique stateIds across multiple ParDos
(1) Do not touch existing RocksDB Store Id with stateId used only by one ParDo
(2) Add a prescan logic in SamzaRunner to traverse Beam pipeline topologically and identify non-unique stateIds.
(3) Using the set of non-unique stateIds, introduce a new mapping of user-provided stateIds to store ID


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@dxichen dxichen merged commit ef20341 into linkedin:li_trunk Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants