Per element schema parsing in ConvertToBeamRows #36393

stankiewicz · 2025-10-04T10:54:03Z

parse message once during transform construction.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…schema is not of tableschema type

gemini-code-assist · 2025-10-04T10:54:14Z

Summary of Changes

Hello @stankiewicz, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces an optimization to the ConvertToBeamRows transform within the BigQuery I/O connector. The change ensures that the BigQuery schema is processed and converted into the appropriate object type only once during the transform's construction. This avoids repetitive schema parsing operations for each data element, leading to improved performance and reduced overhead in data conversion pipelines.

Highlights

Schema Parsing Optimization: The ConvertToBeamRows transform now ensures that the BigQuery schema is parsed only once during its initialization. This prevents redundant schema parsing for every element processed, improving efficiency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

stankiewicz · 2025-10-04T11:45:29Z

R: @ahmedabu98

github-actions · 2025-10-04T11:46:35Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

stankiewicz · 2025-10-06T09:38:50Z

python 3.13 unrelated test failure - apache_beam.examples.wordcount_it_test.WordCountIT

stankiewicz · 2025-10-06T10:37:44Z

python 3.9 = unrelated - AttributeError: 'TestCloudSQLPostgresEnrichment' object has no attribute '_cache_client_retries'

fix here: #36406

stankiewicz · 2025-10-06T11:06:48Z

[gw4] [ 27%] FAILED apache_beam/io/gcp/bigquery_file_loads_test.py::TestBigQueryFileLoads::test_multiple_identical_destinations_on_write_truncate
fix here: #36402

stankiewicz · 2025-10-06T15:25:57Z

Run Python PreCommit 3.13

stankiewicz · 2025-10-06T15:27:41Z

Run Python_ML PreCommit 3.9

stankiewicz · 2025-10-06T15:28:07Z

Run Python_Integration PreCommit 3.13

stankiewicz · 2025-10-06T15:28:22Z

Run Python_Coverage PreCommit 3.9

stankiewicz · 2025-10-06T15:41:31Z

Run Python_ML PreCommit 3.12

stankiewicz · 2025-10-06T15:42:04Z

Run Python_PVR_Flink PreCommit

codecov · 2025-10-06T16:35:18Z

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.87%. Comparing base (9cd994e) to head (f858528).
⚠️ Report is 68 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/io/gcp/bigquery.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #36393      +/-   ##
============================================
+ Coverage     56.84%   56.87%   +0.03%     
  Complexity     3386     3386              
============================================
  Files          1220     1220              
  Lines        185898   186139     +241     
  Branches       3523     3523              
============================================
+ Hits         105672   105875     +203     
- Misses        76885    76923      +38     
  Partials       3341     3341

Flag	Coverage Δ
python	`80.94% <0.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

stankiewicz · 2025-10-06T17:24:43Z

Unrelated errors.

R: @liferoad

liferoad · 2025-10-06T17:29:48Z

/gemini review

liferoad · 2025-10-06T17:31:16Z

sdks/python/apache_beam/io/gcp/bigquery.py

    def __init__(self, schema, dynamic_destinations):
+      if not isinstance(schema,
+                        (bigquery.TableSchema, bigquery.TableFieldSchema)):
+        schema = bigquery_tools.get_bq_tableschema(schema)


do we need remove the lines in beam_row_from_dict?

or beam_row_from_dict is used elsewhere, so we should keep it. Just a thought.

I could remove it but
beam_row_from_dict is publicly used method so removing this if would be potentially breaking change.

gemini-code-assist

Code Review

This pull request introduces a valuable optimization to ConvertToBeamRows by parsing the schema once during the transform's construction rather than for each element. This change should improve performance, especially for large PCollections. The implementation is correct and addresses the goal of the PR. I've added one comment regarding a now-redundant check that could be removed to improve maintainability.

gemini-code-assist · 2025-10-06T17:32:05Z

sdks/python/apache_beam/io/gcp/bigquery.py

+      if not isinstance(schema,
+                        (bigquery.TableSchema, bigquery.TableFieldSchema)):
+        schema = bigquery_tools.get_bq_tableschema(schema)


This is a good optimization to parse the schema once at transform construction. This change makes the schema parsing logic inside bigquery_tools.beam_row_from_dict redundant. To improve maintainability and prevent confusion, consider removing the redundant check from beam_row_from_dict as part of this PR or in a follow-up change.

This reverts commit 1a6ec3a.

…6507) This reverts commit 1a6ec3a.

stankiewicz added 2 commits October 4, 2025 12:51

get_bq_tableschema was invoked on every dict to row conversion if bq …

492cd42

…schema is not of tableschema type

get_bq_tableschema was invoked on every dict to row conversion if bq …

f858528

…schema is not of tableschema type

github-actions bot added python io gcp labels Oct 4, 2025

liferoad approved these changes Oct 6, 2025

View reviewed changes

liferoad reviewed Oct 6, 2025

View reviewed changes

gemini-code-assist bot reviewed Oct 6, 2025

View reviewed changes

liferoad merged commit 1a6ec3a into apache:master Oct 6, 2025
146 of 166 checks passed

This was referenced Oct 14, 2025

The PostCommit Python Xlang Gcp Direct job is flaky #35285

Closed

The PostCommit Python Xlang Gcp Dataflow job is flaky #34990

Closed

Abacn added a commit that referenced this pull request Oct 14, 2025

Revert "Per element schema parsing in ConvertToBeamRows (#36393)"

1448786

This reverts commit 1a6ec3a.

Abacn mentioned this pull request Oct 14, 2025

Revert "Per element schema parsing in ConvertToBeamRows" #36507

Merged

Abacn added a commit that referenced this pull request Oct 14, 2025

Revert "Per element schema parsing in ConvertToBeamRows (#36393)" (#3…

75eda20

…6507) This reverts commit 1a6ec3a.

stankiewicz mentioned this pull request Oct 20, 2025

[python] add setup to BigQuery's convert row Map transform #36502

Merged

3 tasks

Per element schema parsing in ConvertToBeamRows #36393

Per element schema parsing in ConvertToBeamRows #36393

Conversation

stankiewicz commented Oct 4, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Oct 4, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

stankiewicz commented Oct 4, 2025

Uh oh!

github-actions bot commented Oct 4, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stankiewicz commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

codecov bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stankiewicz commented Oct 6, 2025

Uh oh!

liferoad commented Oct 6, 2025

Uh oh!

liferoad Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

liferoad Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

stankiewicz Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

stankiewicz Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stankiewicz commented Oct 6, 2025 •

edited

Loading

stankiewicz commented Oct 6, 2025 •

edited

Loading

codecov bot commented Oct 6, 2025 •

edited

Loading