[Bug]: Can not use Avro over 1.8.2 with Beam 2.52.0

[bvolpato]

What happened?

After #27851, user code that depends on versions newer than Avro 1.8.2 are having problems running on Dataflow.

For example in https://github.com/GoogleCloudPlatform/DataflowTemplates, where we moved on to Avro 1.11.3, there were incompatibility errors:

Caused by: java.io.InvalidClassException: org.apache.avro.specific.SpecificRecordBase; local class incompatible: stream classdesc serialVersionUID = -1463700717714793795, local class serialVersionUID = 189988654766568477

and

Caused by: java.lang.NoSuchMethodError: 'boolean org.apache.avro.generic.GenericRecord.hasField(java.lang.String)' com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.getMetadataIsDeleted(FormatDatastreamRecordToJson.java:258) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:123) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:51) org.apache.beam.sdk.extensions.avro.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:610)

The root cause is that Avro classes are now being shipped along with the /opt/apache/beam/jars/beam-sdks-java-harness.jar, which wasn't the case before.

Tried to relocate in #29407 but got some test failures.

Next step is marking Avro as provided in that JAR, since it's apparently not used.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[bvolpato]

@damccorm I would like to make this a blocker for 2.52.0 (sent an email to dev list https://lists.apache.org/thread/mryy2x4m3f0n1dy3k22fj03rom8x0pot)

[aromanenko-dev]

I'm wondering if there are any tests that are failing because of this?

#27851 was merged about a month ago and I didn't notice any complaints or issues on this despite that it was announced on dev@ ML.

[bvolpato]

I think we should be able to introduce an integration test that relies on a newer Avro version. I can look into it.

The problem here is that for Beam, everything is 1.8.2, so there's no conflict. The issues happened when we tried to run user code that depended on 1.11.3 along with the harness JAR containing 1.8.2, so things ended up conflicting (in our case of Dataflow Runner, always failing because the runner adds the harness at the beginning of the java -cp).

[tvalentyn]

Can this be closed or moved to next milestone?

[bvolpato]

There's an AI to add testing that would have caught this regression -- it's no longer a blocker for 2.52.0 though

[Abacn]

obsolete as beam upgraded to avro 1.11.x