[SUPPORT]: Build failing on latest master on M1 mac due to protobuf support

[pratyakshsharma]

Tips before filing an issue

• Have you gone through our FAQs? yes

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

Tried building the latest master using the command mvn clean install -DskipTests. It fails with the below error -
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (proto-compile) on project hudi-utilities_2.11: Error extracting protoc for version 3.21.5: Unsupported platform: protoc-3.21.5-osx-x86_64.exe -> [Help 1]

To Reproduce

Steps to reproduce the behavior:

1. Use M1 mac
2. Try building the project using mvn clean install -DskipTests

Expected behavior

Build should finish successfully.

[pratyakshsharma]

@codope @nsivabalan @the-other-tim-brown

I see the changes were introduced in this PR - #6135.

I tried few ways to fix this, but the error does not seem to resolve. This is actually hampering my other PRs where hudi-utilities module is getting modified.

[codope]

Looks like protoc-jar-maven-plugin is not yet support for mac osx arm64 architecture. There is a fix but it has not been merged: os72/protoc-jar-maven-plugin#99

[codope]

@the-other-tim-brown any workaround for this issue?

[pratyakshsharma]

As per this link - https://repo1.maven.org/maven2/com/google/protobuf/protoc/3.17.3/, we have the .exe file for M1 mac. I tried degrading the protoc version to 3.17.3 and got the below error -

[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (proto-compile) on project hudi-utilities_2.11: Execution proto-compile of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed.: NullPointerException -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (proto-compile) on project hudi-utilities_2.11: Execution proto-compile of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed. at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293) at org.apache.maven.cli.MavenCli.main (MavenCli.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution proto-compile of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293) at org.apache.maven.cli.MavenCli.main (MavenCli.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: java.lang.NullPointerException at com.github.os72.protocjar.Protoc.extractStdTypes (Protoc.java:372) at com.github.os72.protocjar.maven.ProtocJarMojo.performProtoCompilation (ProtocJarMojo.java:397) at com.github.os72.protocjar.maven.ProtocJarMojo.execute (ProtocJarMojo.java:374) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293) at org.apache.maven.cli.MavenCli.main (MavenCli.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

[the-other-tim-brown]

I see that there was a fix for m1 macs here https://github.com/apache/hudi/pull/6535/files. We can also try another popular proto plugin https://www.xolstice.org/protobuf-maven-plugin/

[jonvex]

I just tried building on master and it worked for me.

[pratyakshsharma]

@jonvex which machine are you using? Is it an M1 mac?

[pratyakshsharma]

@jonvex You mention in this comment - #6535 (comment) that if processor is aarch64, what do you mean by that? Does it have anything to do with the java architecture as well? I would like to know the java version and architecture that you are using. I am actually using x86_64 java 8 as is visible in the output below -

1.8.0_311 (x86_64) "Oracle Corporation" - "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_311.jdk/Contents/Home

Can this be the reason behind the build failing?
I tried on the latest master and using the fix in this PR - #6535. Also tried invalidating caches on Intellij if that might help at all. Nothing seems to work so far.

[jonvex]

I'm using an M1 mac. My java version is:
openjdk version "1.8.0_345" OpenJDK Runtime Environment (Zulu 8.64.0.19-CA-macos-aarch64) (build 1.8.0_345-b01) OpenJDK 64-Bit Server VM (Zulu 8.64.0.19-CA-macos-aarch64) (build 25.345-b01, mixed mode)
You can download it from:
https://www.azul.com/downloads/?package=jdk

[pratyakshsharma]

Downloaded the arm64 java 8 from the link you suggested. The build still fails with the error -

[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (proto-compile) on project hudi-utilities_2.11: Error extracting protoc for version 3.21.5: Unsupported platform: protoc-3.21.5-osx-aarch_64.exe

[slachiewicz]

i've double checked and it works on my computer and I can build latest master without issues.

mvn -v
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/homebrew/Cellar/maven/3.8.6/libexec
Java version: 1.8.0_345, vendor: Azul Systems, Inc., runtime: /Users/slachiewicz/Library/Java/JavaVirtualMachines/azul-1.8.0_345/Contents/Home/jre
Default locale: en_PL, platform encoding: UTF-8
OS name: "mac os x", version: "12.6", arch: "aarch64", family: "mac"

please check Maven settings like enabled profiles - maybe you have some overwrites ?

[slachiewicz]

Maybe you have Rosseta installed and you use Java for x86 and not one dedicated to aarm64? Please paste Yours output of mvn -v