Pin avro 1.12.0 in expansion service #36092
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ahmedabu98, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses a critical dependency conflict by updating the Avro version used within the expansion service. The change ensures compatibility with other project modules, specifically Iceberg, which relies on a different Avro version. This resolution is vital for maintaining the stability and correct functioning of Python and YAML-based data processing pipelines.
Highlights
- Avro Version Pinning: The Avro dependency in the
expansion-servicehas been explicitly pinned to version1.12.0. This change is crucial for resolving a version conflict with theicebergmodule, which requires this specific Avro version, and addresses reported issues affecting Python/YAML pipelines. - Test Suite Trigger: A minor, trivial modification was made to the
.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.jsonfile. This change serves solely to trigger the associatedbeam_PostCommit_Python_Xlang_IO_Directtest suite, ensuring comprehensive testing of the updated dependency.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
LGTM |
| // cannot upgrade this to the latest version due to https://github.com/apache/beam/issues/34993 | ||
| resolutionStrategy.force 'org.apache.avro:avro:1.11.4' | ||
| // iceberg needs avro:1.12.0 | ||
| resolutionStrategy.force 'org.apache.avro:avro:1.12.0' |
There was a problem hiding this comment.
Apache Avro 1.12.0 does not support Java 8. Will this cause the issue for our expansion service?
There was a problem hiding this comment.
We already worked through this when upgrading Iceberg (which also drops Java 8). Note we specify requireJavaVersion: JavaVersion.VERSION_11 at the top
There was a problem hiding this comment.
Expansion service container already uses Java 11. So this should not be an issue.
|
Note that this also results in Iceberg failing for Dataflow Java. |
|
fixes #34993 |
|
It doesn't fix #34993 just yet. We'd need to update avro in BeamModulePlugin to have all of Beam use avro 1.12.0: beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Lines 677 to 678 in a2f9fb8 |
Cherry-picking #36092 into release branch
4 participants
Fixes #32603
Avro was upgraded in iceberg module in #35981, but IO expansion service pins it to 1.11.4, causing a problem for Python/YAML pipelines