Relocate Avro dependencies in Fn Harness JAR

[bvolpato]

After #27851, user code that depends on versions newer than Avro 1.8.2 are having problems running on Dataflow.

For example in https://github.com/GoogleCloudPlatform/DataflowTemplates, where we moved on to Avro 1.11.3, there were incompatibility errors:

Caused by: java.io.InvalidClassException: org.apache.avro.specific.SpecificRecordBase; local class incompatible: stream classdesc serialVersionUID = -1463700717714793795, local class serialVersionUID = 189988654766568477

and

Caused by: java.lang.NoSuchMethodError: 'boolean org.apache.avro.generic.GenericRecord.hasField(java.lang.String)' com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.getMetadataIsDeleted(FormatDatastreamRecordToJson.java:258) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:123) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:51) org.apache.beam.sdk.extensions.avro.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:610)

The root cause is that Avro classes are now being shipped along with the /opt/apache/beam/jars/beam-sdks-java-harness.jar, which wasn't the case before.

Relocating org.apache.avro to org.apache.beam.fn.harness.repackaged.org.apache.avro will solve the problem and allow users to control their Avro.

[bvolpato]

@aromanenko-dev, was the inclusion of Avro intentional in the harness JAR?

@damccorm @Abacn I think this is a blocker for 2.52.0, as there will be pipeline failures if someone depends on anything after Avro 1.8.2 (which was the case for Dataflow Templates, and would require code refactoring).

[bvolpato]

Run Java PreCommit

[bvolpato]

Run Java PreCommit

[bvolpato]

Java PreCommit is getting:

13:19:07 > Configure project :sdks:java:container:java21
13:19:07 docker-pull-licenses set, creating go-licenses
13:19:17 System Go installation: /snap/bin/go is go version go1.16.15 linux/amd64; Preparing to use /home/jenkins/go/bin/go1.21.4
13:19:17 /home/jenkins/go/pkg/mod/golang.org/dl@v0.0.0-20231107201224-8165dcd93058/go1.21.4/main.go:20:8: //go:build comment without // +build comment
13:19:17 
13:19:17 FAILURE: Build failed with an exception.
13:19:17 
13:19:17 * What went wrong:
13:19:17 Could not determine the dependencies of task ':sdks:java:container:goBuild'.
13:19:17 > Could not create task ':sdks:java:container:goPrepare'.
13:19:17    > Process 'command 'sh'' finished with non-zero exit value 1

Which I think is unrelated to the change.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[bvolpato]

Run Java PreCommit

[bvolpato]

Run Java PreCommit

[bvolpato]

Closing in favor of #29412