Skip to content

Per element schema parsing in ConvertToBeamRows #36393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liferoad merged 2 commits into from
Oct 6, 2025
+3 −0
Merged

Per element schema parsing in ConvertToBeamRows #36393

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
492cd42
get_bq_tableschema was invoked on every dict to row conversion if bq …
stankiewicz Oct 4, 2025
f858528
get_bq_tableschema was invoked on every dict to row conversion if bq …
stankiewicz Oct 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions sdks/python/apache_beam/io/gcp/bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2764,6 +2764,9 @@ def expand(self, input):

class ConvertToBeamRows(PTransform):
def __init__(self, schema, dynamic_destinations):
if not isinstance(schema,
(bigquery.TableSchema, bigquery.TableFieldSchema)):
schema = bigquery_tools.get_bq_tableschema(schema)
Copy link
Contributor

@liferoad liferoad Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need remove the lines in beam_row_from_dict?

Copy link
Contributor

@liferoad liferoad Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or beam_row_from_dict is used elsewhere, so we should keep it. Just a thought.

Copy link
Contributor Author

@stankiewicz stankiewicz Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could remove it but
beam_row_from_dict is publicly used method so removing this if would be potentially breaking change.

Comment on lines +2767 to +2769
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This is a good optimization to parse the schema once at transform construction. This change makes the schema parsing logic inside bigquery_tools.beam_row_from_dict redundant. To improve maintainability and prevent confusion, consider removing the redundant check from beam_row_from_dict as part of this PR or in a follow-up change.

Copy link
Contributor Author

@stankiewicz stankiewicz Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good bot

self.schema = schema
self.dynamic_destinations = dynamic_destinations

Expand Down
Loading