[Task]: [Avro] Stop using Avro in "core"

[aromanenko-dev]

What needs to happen?

After extracting all Avro-related class into a separate Avro extension and make all other modules, that leverage Avro in some way, depend on this extension, we need to make sure that "core" doesn't rely on Avro anymore.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[bvolpato]

Hi @aromanenko-dev! Thanks for the effort here!

Do we have any plans to move this forward? We're noticing concerns regarding vulnerabilities/CVEs on jackson-mapper-asl (which comes from Avro 1.8.2) -- so wondering if we should put any effort on upgrading that dependency, or if it's best to proceed here and get rid of it in "core".

[aromanenko-dev]

Hi @bvolpato !

Actually, this task is suspended by the policy that we have for breaking changes in Beam. This one is a part of epic task #24292 where we did quite a lot of progress by introducing a new Avro extension for Java SDK (it was tested and finally should support different Avro versions), transitioning all other modules, that leverages Avro from core, to this extension and deprecating Avro-related classed in core. Though, we can't just drop an Avro support from core to avoid instant breaking changes, so we need to wait several minor Beam releases (at least three, iirc) and give the user some time to switching and test this new extension.

Do you think you can also start using extensions/avro with a more recent Avro version and in this case I think 1.8.2 will be dropped from classpath and won't bring the other out-dated dependencies?

[bvolpato]

Thanks for the response @aromanenko-dev! I'll look into using extensions/avro. Although vulnerability scanners will still dislike having Avro 1.8.2 packaged, it sounds like we have a good plan to move forward and eventually get rid of it.

Hi @aromanenko-dev, I went over most of the Beam issues and PRs related to the avro dependency (dated back from 2019!) and have now reached this task. Thank you again for leading the entire epic task!

I have two questions to ask if you don't mind.

• I notice we still have avro versions defined in
  https://github.com/apache/beam/blob/v2.48.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L589-L590. Are we planning to clean them up?
• Do we have a timeline for when the remaining work will be completed and released? I have included some upcoming releases and cut-off dates below for your reference:
  2.50.0: Aug 9, 2023
  2.51.0: Sep 20, 2023

[aromanenko-dev]

@shunping-google
Thanks, I'm happy to help!

1. We will update Avro version to more recent one.
2. We deprecated Avro in "core" in Beam 2.46.0 and we usually remove deprecated stuff, at least, after 3 Beam releases if there are no issues discovered during this period. So, for the moment, I don't see any blockers to remove it in, say, 2.50.0. Anyway, I'll send an email to mailing list before to make sure that there are no other objections for this.