Skip to content

[Feature Request]: Enable Avro GenericRecord as a valid type for BigQuery StorageWrite API ingestion #24329

@prodriguezdefino

Description

@prodriguezdefino

What would you like to happen?

When using AVRO GenericRecord as the transport object in their pipelines, users need to configure the use of Beam schemas in the BigQueryIO to transform their data into Beam Rows first, or provide a format function to transform into TableRow, to then have their data formatted as DynamicMessage protos for BQ StorageWrite to work.

By using GenericRecord as the input type for the PCollection, Beam should be able to translate the input into StorageWrite proto payload to save non-needed format transformations (to BeamRow or to TableRow before becoming proto), potentially improving performance and saving some CPU cycles.

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

Metadata

Metadata

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions