[DO NOT MERGE] Cherrypick pr 35817 #35826
Conversation
…e#35817) * fix(bigquery): use correct project for temp dataset operations Add _get_temp_dataset_project helper method to determine the correct project ID for temporary dataset operations. Update _setup_temporary_dataset and finish_bundle to use this method instead of _get_project when cleaning up or creating temporary datasets. This ensures the correct project is used when temp_dataset is a DatasetReference. Add comprehensive unit tests to verify the behavior with different temp_dataset configurations. * test(bigquery): handle missing bigquery dependencies in tests Skip tests when BigQuery dependencies are not installed to prevent test failures * fix lint * feat(bigquery): add temp table project resolution helper Add _get_temp_table_project method to handle project ID resolution for temporary tables Add corresponding tests to verify fallback behavior
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @liferoad, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request is a cherry-pick of an upstream change (PR 35817) into Beam 2.61.0, primarily for building a patched version. The core change addresses an issue in Apache Beam's BigQuery I/O connector where temporary BigQuery datasets and tables might not always be created or cleaned up in the correct Google Cloud project. I've introduced logic to ensure that when a specific DatasetReference or TableReference is provided for temporary resources, their associated project ID is correctly used for creation and cleanup, rather than always defaulting to the pipeline's project. This enhances flexibility and correctness for BigQuery operations involving temporary resources across different projects.
Highlights
- Enhanced Project Resolution for Temporary BigQuery Resources: I've introduced new helper methods (
_get_temp_dataset_projectand_get_temp_table_project) within the BigQuery I/O components. These methods are responsible for intelligently determining the correct Google Cloud project ID to use when creating or interacting with temporary BigQuery datasets and tables. - Corrected Temporary Resource Management in BigQuery I/O: I've updated the BigQuery read and write operations to leverage these newly introduced project resolution methods. This ensures that temporary datasets and tables, which are often used for intermediate query results or staging, are consistently created and cleaned up within the project explicitly specified by a
DatasetReferenceorTableReference, rather than always defaulting to the pipeline's project. - Comprehensive Unit Test Coverage: To validate the new logic, I've added comprehensive unit tests. These tests cover various scenarios for specifying temporary dataset and table projects, including using string IDs,
DatasetReferenceobjects, andValueProviderinstances, ensuring the correct project is always identified and used for BigQuery operations.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
1 participant
Only to build the patched beam 2.61.0 to include #35817
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.