BigQuerySamplesIT.testTableIO is breaking in Java PostCommits

[ahmedabu98]

What happened?

beam_PostCommit_Java_DataflowVX and beam_PostCommit_Java_Dataflow_Examples_Dataflow_V2_X are failing due to the following exception:
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.VerifyException: Expected Decimal logical type

Issue Priority

Priority: 1

Issue Component

Component: io-java-gcp

[ahmedabu98]

An error is thrown when reading in a Numeric BigQuery field with EXPORT method. Reproduced this by reading from a BQ table that has a single column of type Numeric:

import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;

public class SimpleRead {
  public static void main(String[] args) {
    String mytable = "<my table>";
    PipelineOptions options = PipelineOptionsFactory.create();
    options.setTempLocation("<my temp GCS bucket>");
    Pipeline p = Pipeline.create(options);

    p
        .apply(BigQueryIO.readTableRows()
            .from(mytable));
    p.run().waitUntilFinish();
  }
}

[ahmedabu98]

The BigQueryAvroUtils.toGenericAvroSchema() method used here returns a null "type" value for the Numeric type:

{"type":"record","name":"root","namespace":"org.apache.beam.sdk.io.gcp.bigquery","doc":"Translated Avro Schema for root","fields":[{"name":"numeric_field","type":["null","bytes"]}]}

[ahmedabu98]

The schema was previously generated with BigQueryIO.JSON_FACTORY but was changed here. I added the method back and logged the schema results:

{"fields":[{"mode":"NULLABLE","name":"numeric_field","type":"NUMERIC"}]}

[steveniemitz]

@kkdoon

[kkdoon]

Thanks @ahmedabu98 for opening this issue. The root cause is the reader schema generated by BigQueryAvroUtils.toGenericAvroSchema() for numeric BigQuery type, where the expected schema should be like:{"type":"bytes","logicalType":"decimal","precision":38,"scale":9}, however this function generates {"type":"bytes"} instead. Using only the writer schema fixes this issue since it reads the correct AVRO schema.

I have created a PR #23594 for the same.