sync with open source how #118
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.255.0 to 0.256.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.255.0...v0.256.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.256.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Reuse secret instead of creating every time * Reuse secret instead of creating every time * lint * lint
…d record in output builder
…ords (#36459) * Fix CassandraIO ReadFn to quote column names for reserved keywords * Minor fixes
* Add new method of generating key for GBEK * Java version * fix deps * Imports * Secret parsing tests * docs * more docs * formatting + test cleanup * lint * lint * lint * lint * import order * Deps + style exemption * reuse key: * reuse key * Feedback * Test fixes
* WIP - copy-pasta v9.x tests as a starting point * Updates tests to be compatible with ES v9 Bulk API error message format * Adds es 9 to pre-commit workflow * Adds clientkeystore for elasticsearch-9 tests * Adds note to CHANGES.md
* Revert #36949 * Update _MongoSink
This reverts commit f242a88.
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.256.0 to 0.257.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.256.0...v0.257.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.257.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Makes IntervalWindowCoder::isRegisterByteSizeObserverCheap directly return true. Add getEncodedElementByteSize to PaneInfoCoder Add registerByteSizeObserver to ValueWithRecordIdCoder
Fix Republish Released Docker Images job
…#36889) * Update build.gradle Enables some persistent state type validates runner tests. * Update beam_PostCommit_Java_ValidatesRunner_Dataflow_V2_Streaming.json * Update beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.json * Update beam_PostCommit_Java_ValidatesRunner_Dataflow.json * Update beam_PostCommit_Java_ValidatesRunner_Dataflow_V2.json
) Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.40.0 to 1.40.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@v1.40.0...v1.40.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.40.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Revert "Include Beam version details in MongoDB handshake"
* Add LRU cache eviction to CachingStateProvider Fixes #37213 Implements LRU (Least Recently Used) cache eviction to prevent unbounded memory growth in long-running workers. Adds configurable maxCacheSize parameter (default: 1000 entries) and maintains LRU order using JavaScript Map's insertion order. - Add maxCacheSize constructor parameter with default value of 1000 - Implement evictIfNeeded() to remove oldest entry when cache is full - Implement touchCacheEntry() to move accessed items to end (LRU) - Add comprehensive test coverage in state_provider_test.ts This addresses the TODO comment in the code and improves reliability for production workloads. * Address review comments: size-based LRU eviction for CachingStateProvider - Fixed bug: removed incorrect evictIfNeeded() call in promise callback - Removed unnecessary this_ variable (arrow functions capture this) - Changed from count-based to size-based eviction (similar to Python statecache.py) - Added estimateSize() to calculate memory weight of cached values - Default cache weight: 100MB - Updated tests to work with weight-based eviction * Fix prettier formatting * Address review comments: circular references, eviction ordering, tests - Fixed sizeof function to handle circular references using visited Set - Fixed eviction ordering: add to cache first, then evict (fixes edge case) - Added test for oversized item that exceeds maxCacheWeight - Implemented custom sizeof instead of object-sizeof package (has Node.js compatibility issues) * Address Gemini comments: fix race condition, optimize evictIfNeeded - Fixed critical race condition in promise callback: only update cache if the entry is still the same promise we're resolving - Optimized evictIfNeeded: use entries() iterator and removed redundant checks
…37325) Bumps [keras](https://github.com/keras-team/keras) from 3.12.0 to 3.13.1. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](keras-team/keras@v3.12.0...v3.13.1) --- updated-dependencies: - dependency-name: keras dependency-version: 3.13.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This reverts commit 8da0802.
* Add Terraform files to deploy Envoy RateLimiter * fix variables * Add nat creation command to readme * add hpa for memory * fix redability comments * fix comments * add license * Update examples/terraform/envoy-ratelimiter/README.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
…d has a flaky test. (#37317)
Co-authored-by: damccorm <actions@main-runner-vplcm-927dk.local>
Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.20.19 to 1.21.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.20.19...v1.21.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* porting and some improvements * fix some publisher issues * checkstyle, import, etc changes * add changes info on datadog * updated formatting and correct issue link * fix format issue * fix comments * add precommit for datadog * fix nullable annotation
…7366) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.14.1 to 2.15.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Commits](linkedin/goavro@v2.14.1...v2.15.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: damccorm <actions@GitHub Actions 1010719742.local>
* Migrate Cloudbuild usage to Python client * linting * slide cloud build into main deoendency * Base image requirement updates * attempt to make google.api_core a non-default dependency * yapf
* Update website for 2.71.0 release * Update website/www/site/content/en/blog/beam-2.71.0.md Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com> * Update CHANGES.md * Update release date for version 2.71.0 * Update download link for Beam 2.71.0 release --------- Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
* [#37209] Enhance serialization error messages for better DX Improved error messages when user code fails to serialize (pickle) for distributed execution. The original error was too technical and didn't explain the cause or suggest fixes. Changes: - Enhanced RuntimeError message with clear explanation of why serialization is required - Added common causes (lambdas capturing file handles, DB connections, thread locks) - Provided three concrete fixes: module-level functions, setup() methods, checking closure captures - Broadened exception catching to include TypeError and other pickling failures (not just RuntimeError) - Added exception chaining (from e) to preserve original stack trace - Added test case to verify the new error message content This significantly improves developer experience when debugging serialization issues, especially for new Apache Beam users. Fixes #37209 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Apply yapf formatting Fix Python formatter precommit check by applying yapf v0.43.0 formatting rules to modified files. * [#37209] Make withBackOffSupplier public to enable bounded retry configuration Users need to configure bounded backoff to prevent infinite retry loops. Making withBackOffSupplier public allows users to set FluentBackoff.DEFAULT.withMaxRetries(n) and control retry behavior. Added integration test demonstrating bounded retry with maxRetries=3. Related to #37198, #37176 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Revert unrelated Java changes --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Undo #35962 and move from distutils to setuptools instead
* Make a copy of GcsUtil.java as GcsUtilLegacy.java * Change references from GcsUtil to GcsUtilLegacy in GcsUtilLegacy.java * Change public apis in GcsUtil.java to call the GcsUtilLegacy.java * Add mocking GcsUtilLegacy for tests and remove unnecessary references of gcsio in GcsUtil * Possible breaking change: removed unused public functions. * Get rid of the last bit (read option) gcsio references in GcsUtil.java. * Route the code path of open reading files to include api count metric. * Spotless check * Remove BatchInterface and RewriteOp from GcsUtil.java. Simplify some functions. * Rename GcsUtilLegacy to GcsUtilV1 * Make GcsUtilV1 package private * Import ExperimentalOptions * Move GcsReadOptionsFactory from GcsUtilV1 to GcsOptions * Fix a failed test * Add comments to SupressWarnings * Fix test
* remove ReadWrite PubsubLite Yaml * add back pubsub oops * and the configuration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.