[Failing Test]: Python PostCommit failing due to duplicate AvroSchemaIO autoservice

[Abacn]

What happened?

Python PostCommit jdbcio_xlang_it_test failing. Jenkins log does not show much message, but running locally the actual error shows:

ERROR:apache_beam.utils.subprocess_server:Starting job service with ['java', '-jar', '/Users/yathu/.apache_beam/cache/jars/composite-jars/9b6e2cb01bd723cbd87a5e71462f25664f165993697afc98e1c64c34bf814f98.jar', '50709', '--filesToStage=/Users/yathu/dev/virtualenv/py38beam/lib/sdks/java/extensions/schemaio-expansion-service/build/libs/beam-sdks-java-extensions-schemaio-expansion-service-2.47.0-SNAPSHOT.jar,/Users/yathu/.apache_beam/cache/jars/postgresql-42.2.16.jar']
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
  File "/Users/yathu/dev/virtualenv/py38beam/lib/python3.8/site-packages/apache_beam/utils/subprocess_server.py", line 88, in start
    raise RuntimeError(
RuntimeError: Service failed to start up with error 1
Traceback (most recent call last):

This is because there are two @AutoService AvroSchemaIOProvider classes so the schemaio expansion service fails to start.

The fix could either be similar to the workaround here:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/Providers.java

Line 45 in 011296c

// Avro provider is treated as a special case since two Avro providers may want to be loaded -

Or simply remove AvroSchemaIOProvider in core.

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

CC: @aromanenko-dev

[damccorm]

@aromanenko-dev this is currently the only known release blocker - do you think you'll be able to get a fix in quickly, or should we consider temporarily reverting #25534 (since if I understand correctly, that is the cause of the failures)?

[mosche]

@Abacn @aromanenko-dev I was just looking at this as well. I agree with @aromanenko-dev that removing anything from core might be problematic. Though, also, it turns out that org.apache.beam.sdk.extensions.avro.io.AvroSchemaIOProvider isn't even part of the uber expansion jar, there's a separate one build for testing (:sdks:java:testing:expansion-service)

[Abacn]

another quick fix for 2.46.0 would be remove the SchemaIOProvider AutoService in extension-avro, or even just comment out the @AutoService annotation. This would not introduce any breaking change and also let expansion service working properly